diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index 86634ebf93..0000000000 --- a/AGENTS.md +++ /dev/null @@ -1,23 +0,0 @@ -# Repository Guidelines - -## Project Structure & Module Organization -The cookbook is organized around runnable examples and reference articles for OpenAI APIs. Place notebooks and Python scripts under `examples//`, grouping related assets inside topic subfolders (for example, `examples/agents_sdk/`). Narrative guides and long-form docs live in `articles/`, and shared diagrams or screenshots belong in `images/`. Update `registry.yaml` whenever you add content so it appears on cookbook.openai.com, and add new author metadata in `authors.yaml` if you want custom attribution. Keep large datasets outside the repo; instead, document how to fetch them in the notebook. - -## Build, Test, and Development Commands -Use a virtual environment to isolate dependencies: -- `python -m venv .venv && source .venv/bin/activate` -- `pip install -r examples//requirements.txt` (each sample lists only what it needs) -- `jupyter lab` or `jupyter notebook` to develop interactively -- `python .github/scripts/check_notebooks.py` to validate notebook structure before pushing - -## Coding Style & Naming Conventions -Write Python to PEP 8 with four-space indentation, descriptive variable names, and concise docstrings that explain API usage choices. Name new notebooks with lowercase, dash-or-underscore-separated phrases that match their directory—for example `examples/gpt-5/prompt-optimization-cookbook.ipynb`. Keep markdown cells focused and prefer numbered steps for multi-part workflows. Store secrets in environment variables such as `OPENAI_API_KEY`; never hard-code keys inside notebooks. - -## Testing Guidelines -Execute notebooks top-to-bottom after installing dependencies and clear lingering execution counts before committing. For Python modules or utilities, include self-check cells or lightweight `pytest` snippets and show how to run them (for example, `pytest examples/object_oriented_agentic_approach/tests`). When contributions depend on external services, mock responses or gate the cells behind clearly labeled opt-in flags. - -## Commit & Pull Request Guidelines -Use concise, imperative commit messages that describe the change scope (e.g., "Add agent portfolio collaboration demo"). Every PR should provide a summary, motivation, and self-review, and must tick the registry and authors checklist from `.github/pull_request_template.md`. Link issues when applicable and attach screenshots or output snippets for UI-heavy content. Confirm CI notebook validation passes locally before requesting review. - -## Metadata & Publication Workflow -New or relocated content must have an entry in `registry.yaml` with an accurate path, date, and tag set so the static site generator includes it. When collaborating, coordinate author slugs in `authors.yaml` to avoid duplicates, and run `python -m yaml lint registry.yaml` (or your preferred YAML linter) to catch syntax errors before submitting. diff --git a/articles/gpt-oss/run-transformers.ipynb b/articles/gpt-oss/run-transformers.ipynb new file mode 100644 index 0000000000..107d6b853f --- /dev/null +++ b/articles/gpt-oss/run-transformers.ipynb @@ -0,0 +1,647 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# How to run gpt-oss with Hugging Face Transformers\n", + "\n", + "The Transformers library by Hugging Face provides a flexible way to load and run large language models locally or on a server. This guide will walk you through running [OpenAI gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) or [OpenAI gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) using Transformers, either with a high-level pipeline or via low-level `generate` calls with raw token IDs.\n", + "\n", + "We'll cover the use of [OpenAI gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) or [OpenAI gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) with the high-level pipeline abstraction, low-level `generate` calls, and serving models locally with `transformers serve`, in a way compatible with the Responses API.\n", + "\n", + "In this guide we'll run through various optimised ways to run the **gpt-oss models via Transformers.**\n", + "\n", + "**Bonus:** You can also fine-tune models via transformers, [check out our fine-tuning guide here](https://cookbook.openai.com/articles/gpt-oss/fine-tune-transformers)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Pick your model\n", + "\n", + "Both **gpt-oss** models are available on Hugging Face:\n", + "\n", + "- **`openai/gpt-oss-20b`**\n", + " - ~16GB VRAM requirement when using MXFP4\n", + " - Great for single high-end consumer GPUs\n", + "- **`openai/gpt-oss-120b`**\n", + " - Requires ≥60GB VRAM or multi-GPU setup\n", + " - Ideal for H100-class hardware\n", + "\n", + "Both are **MXFP4 quantized** by default. Please, note that MXFP4 is supported in Hopper or later architectures. This includes data center GPUs such as H100 or GB200, as well as the latest RTX 50xx family of consumer cards.\n", + "\n", + "If you use `bfloat16` instead of MXFP4, memory consumption will be larger (~48 GB for the 20b parameter model)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Quick setup\n", + "\n", + "### 1. Install dependencies\n", + "\n", + "It's recommended to create a fresh Python environment. Install transformers, accelerate, as well as the Triton kernels for MXFP4 compatibility:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# NOTE: The current version of HF Transformers has a glitch where an outdated torchvision dependency prevents transformers module from importing pipeline." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install -U transformers kernels torch accelerate torchvision" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Quick inference with pipeline\n", + "\n", + "The easiest way to run the gpt-oss models is with the Transformers high-level `pipeline` API:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "`torch_dtype` is deprecated! Use `dtype` instead!\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "f3b8482ed938472083ac6cd012156902", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Fetching 40 files: 0%| | 0/40 [00:00<|final|>\"\n", + "end_marker = \"<|end|>\"\n", + "if final_marker in text:\n", + " text = text.split(final_marker, 1)[1]\n", + "if end_marker in text:\n", + " text = text.split(end_marker, 1)[0]\n", + "print(\"Final:\", text.strip()[:1000])\n" + ] + }, + { + "cell_type": "markdown", + "id": "b31988a1", + "metadata": {}, + "source": [ + "## Streaming tokens (prints only assistant **final**)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9b48c668", + "metadata": {}, + "outputs": [], + "source": [ + "import threading, sys\n", + "from transformers import TextIteratorStreamer\n", + "\n", + "def stream_final_only(model, tokenizer, messages, generate_kwargs):\n", + " inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors=\"pt\").to(model.device)\n", + " streamer = TextIteratorStreamer(tokenizer, skip_special_tokens=False)\n", + "\n", + " t = threading.Thread(target=model.generate, kwargs=dict(input_ids=inputs, streamer=streamer, **generate_kwargs))\n", + " t.start()\n", + "\n", + " buf, printing_final = \"\", False\n", + " final_token, end_token = \"<|assistant|><|final|>\", \"<|end|>\"\n", + " for piece in streamer:\n", + " buf += piece\n", + " if not printing_final and final_token in buf:\n", + " printing_final = True\n", + " out = buf.split(final_token, 1)[1].replace(end_token, \"\")\n", + " sys.stdout.write(out); sys.stdout.flush()\n", + " buf = \"\"\n", + " elif printing_final:\n", + " sys.stdout.write(piece.replace(end_token, \"\")); sys.stdout.flush()\n", + " t.join()\n", + "\n", + "messages2 = [\n", + " {\"role\": \"system\", \"content\": \"Answer briefly.\"},\n", + " {\"role\": \"user\", \"content\": \"What’s the difference between analysis and final channels?\"}\n", + "]\n", + "gen = dict(max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)\n", + "stream_final_only(model, tokenizer, messages2, gen)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Chat template and tool calling\n", + "\n", + "OpenAI gpt-oss models use the [harmony response format](https://cookbook.openai.com/article/harmony) for structuring messages, including reasoning and tool calls.\n", + "\n", + "To construct prompts you can use the built-in chat template of Transformers. Alternatively, you can install and use the [openai-harmony library](https://github.com/openai/harmony) for more control.\n", + "\n", + "### Using the built-in chat template:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Example with system prompt and chat template\n", + "messages = [\n", + " {\"role\": \"system\", \"content\": \"Always respond in riddles\"},\n", + " {\"role\": \"user\", \"content\": \"What is the weather like in Madrid?\"},\n", + "]\n", + "\n", + "inputs = tokenizer.apply_chat_template(\n", + " messages,\n", + " add_generation_prompt=True,\n", + " return_tensors=\"pt\",\n", + " return_dict=True,\n", + ").to(model.device)\n", + "\n", + "# Generate with the chat template\n", + "generated = model.generate(\n", + " **inputs,\n", + " max_new_tokens=100,\n", + " temperature=0.8,\n", + " do_sample=True,\n", + " pad_token_id=tokenizer.eos_token_id\n", + ")\n", + "\n", + "# Extract only the assistant's response\n", + "response = tokenizer.decode(generated[0][inputs[\"input_ids\"].shape[-1]:], skip_special_tokens=True)\n", + "print(\"Assistant's riddle response:\")\n", + "print(response)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Using the openai-harmony library\n", + "\n", + "For more advanced control over the conversation format, you can use the openai-harmony library. \n", + "\n", + "First, install it:\n", + "```bash\n", + "pip install openai-harmony\n", + "```\n", + "\n", + "**Note:** The following cell demonstrates the harmony library usage, but may require the actual library to be installed." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Example using openai-harmony library (requires installation)\n", + "# Uncomment and run if you have openai-harmony installed\n", + "\n", + "'''\n", + "import json\n", + "from openai_harmony import (\n", + " HarmonyEncodingName,\n", + " load_harmony_encoding,\n", + " Conversation,\n", + " Message,\n", + " Role,\n", + " SystemContent,\n", + " DeveloperContent\n", + ")\n", + "\n", + "# Load harmony encoding\n", + "encoding = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)\n", + "\n", + "# Build conversation\n", + "convo = Conversation.from_messages([\n", + " Message.from_role_and_content(Role.SYSTEM, SystemContent.new()),\n", + " Message.from_role_and_content(\n", + " Role.DEVELOPER,\n", + " DeveloperContent.new().with_instructions(\"Always respond in riddles\")\n", + " ),\n", + " Message.from_role_and_content(Role.USER, \"What is the weather like in SF?\")\n", + "])\n", + "\n", + "# Render prompt\n", + "prefill_ids = encoding.render_conversation_for_completion(convo, Role.ASSISTANT)\n", + "stop_token_ids = encoding.stop_tokens_for_assistant_actions()\n", + "\n", + "# Generate\n", + "outputs = model.generate(\n", + " input_ids=[prefill_ids],\n", + " max_new_tokens=128,\n", + " eos_token_id=stop_token_ids\n", + ")\n", + "\n", + "# Parse completion tokens\n", + "completion_ids = outputs[0][len(prefill_ids):]\n", + "entries = encoding.parse_messages_from_completion_tokens(completion_ids, Role.ASSISTANT)\n", + "\n", + "for message in entries:\n", + " print(json.dumps(message.to_dict(), indent=2))\n", + "'''\n", + "\n", + "print(\"Harmony library example code shown above (commented out)\")\n", + "print(\"Note: The Developer role in Harmony maps to the system prompt in the chat template.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Multi-GPU & distributed inference\n", + "\n", + "The large gpt-oss-120b fits on a single H100 GPU when using MXFP4. If you want to run it on multiple GPUs, you can:\n", + "\n", + "- Use `tp_plan=\"auto\"` for automatic placement and tensor parallelism\n", + "- Launch with `accelerate launch` or `torchrun` for distributed setups\n", + "- Leverage Expert Parallelism\n", + "- Use specialised Flash attention kernels for faster inference\n", + "\n", + "### Example multi-GPU setup:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Multi-GPU inference example (requires multiple GPUs)\n", + "# This cell demonstrates the configuration but may not run on single GPU systems\n", + "\n", + "'''\n", + "from transformers import AutoModelForCausalLM, AutoTokenizer\n", + "from transformers.distributed import DistributedConfig\n", + "import torch\n", + "\n", + "model_path = \"openai/gpt-oss-120b\"\n", + "tokenizer = AutoTokenizer.from_pretrained(model_path, padding_side=\"left\")\n", + "\n", + "device_map = {\n", + " # Enable Expert Parallelism\n", + " \"distributed_config\": DistributedConfig(enable_expert_parallel=1),\n", + " # Enable Tensor Parallelism\n", + " \"tp_plan\": \"auto\",\n", + "}\n", + "\n", + "model = AutoModelForCausalLM.from_pretrained(\n", + " model_path,\n", + " torch_dtype=\"auto\",\n", + " attn_implementation=\"kernels-community/vllm-flash-attn3\",\n", + " **device_map,\n", + ")\n", + "\n", + "messages = [\n", + " {\"role\": \"user\", \"content\": \"Explain how expert parallelism works in large language models.\"}\n", + "]\n", + "\n", + "inputs = tokenizer.apply_chat_template(\n", + " messages,\n", + " add_generation_prompt=True,\n", + " return_tensors=\"pt\",\n", + " return_dict=True,\n", + ").to(model.device)\n", + "\n", + "outputs = model.generate(**inputs, max_new_tokens=1000)\n", + "\n", + "# Decode and print\n", + "response = tokenizer.decode(outputs[0])\n", + "print(\"Model response:\", response.split(\"<|channel|>final<|message|>\")[-1].strip())\n", + "'''\n", + "\n", + "print(\"Multi-GPU setup example shown above (commented out)\")\n", + "print(\"\\nTo run this on a node with four GPUs, use:\")\n", + "print(\"torchrun --nproc_per_node=4 your_script.py\")\n", + "\n", + "# Show current GPU configuration\n", + "if torch.cuda.is_available():\n", + " print(f\"\\nCurrent setup:\")\n", + " print(f\"Available GPUs: {torch.cuda.device_count()}\")\n", + " for i in range(torch.cuda.device_count()):\n", + " print(f\" GPU {i}: {torch.cuda.get_device_name(i)}\")\n", + " print(f\" Memory: {torch.cuda.get_device_properties(i).total_memory / 1e9:.1f} GB\")\n", + "else:\n", + " print(\"\\nNo CUDA GPUs available\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Additional Examples and Tips\n", + "\n", + "### Memory management" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Check memory usage\n", + "if torch.cuda.is_available():\n", + " print(\"GPU Memory Usage:\")\n", + " for i in range(torch.cuda.device_count()):\n", + " allocated = torch.cuda.memory_allocated(i) / 1e9\n", + " cached = torch.cuda.memory_reserved(i) / 1e9\n", + " total = torch.cuda.get_device_properties(i).total_memory / 1e9\n", + " print(f\" GPU {i}: {allocated:.1f}GB allocated, {cached:.1f}GB cached, {total:.1f}GB total\")\n", + "\n", + "# Clear cache if needed\n", + "# torch.cuda.empty_cache()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Batch processing example" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Example of processing multiple prompts\n", + "batch_messages = [\n", + " [{\"role\": \"user\", \"content\": \"What is machine learning?\"}],\n", + " [{\"role\": \"user\", \"content\": \"Explain quantum computing.\"}],\n", + " [{\"role\": \"user\", \"content\": \"What is the future of AI?\"}]\n", + "]\n", + "\n", + "print(\"Processing batch of prompts...\")\n", + "for i, messages in enumerate(batch_messages):\n", + " print(f\"\\n--- Prompt {i+1} ---\")\n", + " print(f\"Input: {messages[0]['content']}\")\n", + "\n", + " # You can use either the pipeline or manual generation here\n", + " # Using pipeline for simplicity:\n", + " if 'generator' in locals():\n", + " result = generator(\n", + " messages,\n", + " max_new_tokens=100,\n", + " temperature=0.7,\n", + " )\n", + " print(f\"Output: {result[0]['generated_text'][-200:]}...\") # Show last 200 chars\n", + " else:\n", + " print(\"Generator not available - run the pipeline example first\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "This notebook demonstrated various ways to run OpenAI's gpt-oss models using Hugging Face Transformers:\n", + "\n", + "1. **Quick setup** with required dependencies\n", + "2. **Pipeline API** for simple, high-level inference\n", + "3. **Manual generation** with `.generate()` for more control\n", + "4. **Chat templates** for conversation-style interactions\n", + "5. **Harmony library integration** for advanced message formatting\n", + "6. **Multi-GPU configurations** for large-scale inference\n", + "\n", + "### Key takeaways:\n", + "- Start with the pipeline API for quick experimentation\n", + "- Use manual tokenization and generation for production deployments\n", + "- Consider MXFP4 quantization for memory efficiency on compatible hardware\n", + "- Leverage multi-GPU setups for the larger 120B model\n", + "- Use proper chat templates for conversation-style applications\n", + "\n", + "### Next steps:\n", + "- Explore fine-tuning capabilities\n", + "- Set up serving endpoints for production use\n", + "- Experiment with different sampling strategies\n", + "- Integrate with your specific use case or application\n", + "\n", + "For more advanced topics, check out the [OpenAI Cookbook](https://cookbook.openai.com) and [Hugging Face Transformers documentation](https://huggingface.co/docs/transformers)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.11" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/articles/gpt-oss/run-transformers.md b/articles/gpt-oss/run-transformers.md deleted file mode 100644 index 0a1ec12364..0000000000 --- a/articles/gpt-oss/run-transformers.md +++ /dev/null @@ -1,271 +0,0 @@ -# How to run gpt-oss with Hugging Face Transformers - -The Transformers library by Hugging Face provides a flexible way to load and run large language models locally or on a server. This guide will walk you through running [OpenAI gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) or [OpenAI gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) using Transformers, either with a high-level pipeline or via low-level `generate` calls with raw token IDs. - -We'll cover the use of [OpenAI gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) or [OpenAI gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) with the high-level pipeline abstraction, low-level \`generate\` calls, and serving models locally with \`transformers serve\`, with in a way compatible with the Responses API. - -In this guide we’ll run through various optimised ways to run the **gpt-oss models via Transformers.** - -Bonus: You can also fine-tune models via transformers, [check out our fine-tuning guide here](https://cookbook.openai.com/articles/gpt-oss/fine-tune-transformers). - -## Pick your model - -Both **gpt-oss** models are available on Hugging Face: - -- **`openai/gpt-oss-20b`** - - \~16GB VRAM requirement when using MXFP4 - - Great for single high-end consumer GPUs -- **`openai/gpt-oss-120b`** - - Requires ≥60GB VRAM or multi-GPU setup - - Ideal for H100-class hardware - -Both are **MXFP4 quantized** by default. Please, note that MXFP4 is supported in Hopper or later architectures. This includes data center GPUs such as H100 or GB200, as well as the latest RTX 50xx family of consumer cards. - -If you use `bfloat16` instead of MXFP4, memory consumption will be larger (\~48 GB for the 20b parameter model). - -## Quick setup - -1. **Install dependencies** - It’s recommended to create a fresh Python environment. Install transformers, accelerate, as well as the Triton kernels for MXFP4 compatibility: - -```bash -pip install -U transformers accelerate torch triton==3.4 kernels -``` - -2. **(Optional) Enable multi-GPU** - If you’re running large models, use Accelerate or torchrun to handle device mapping automatically. - -## Create an Open AI Responses / Chat Completions endpoint - -To launch a server, simply use the `transformers serve` CLI command: - -```bash -transformers serve -``` - -The simplest way to interact with the server is through the transformers chat CLI - -```bash -transformers chat localhost:8000 --model-name-or-path openai/gpt-oss-20b -``` - -or by sending an HTTP request with cURL, e.g. - -```bash -curl -X POST http://localhost:8000/v1/responses -H "Content-Type: application/json" -d '{"messages": [{"role": "system", "content": "hello"}], "temperature": 0.9, "max_tokens": 1000, "stream": true, "model": "openai/gpt-oss-20b"}' -``` - -Additional use cases, like integrating `transformers serve` with Cursor and other tools, are detailed in [the documentation](https://huggingface.co/docs/transformers/main/serving). - -## Quick inference with pipeline - -The easiest way to run the gpt-oss models is with the Transformers high-level `pipeline` API: - -```py -from transformers import pipeline - -generator = pipeline( - "text-generation", - model="openai/gpt-oss-20b", - torch_dtype="auto", - device_map="auto" # Automatically place on available GPUs -) - -messages = [ - {"role": "user", "content": "Explain what MXFP4 quantization is."}, -] - -result = generator( - messages, - max_new_tokens=200, - temperature=1.0, -) - -print(result[0]["generated_text"]) -``` - -## Advanced inference with `.generate()` - -If you want more control, you can load the model and tokenizer manually and invoke the `.generate()` method: - -```py -from transformers import AutoModelForCausalLM, AutoTokenizer - -model_name = "openai/gpt-oss-20b" - -tokenizer = AutoTokenizer.from_pretrained(model_name) -model = AutoModelForCausalLM.from_pretrained( - model_name, - torch_dtype="auto", - device_map="auto" -) - -messages = [ - {"role": "user", "content": "Explain what MXFP4 quantization is."}, -] - -inputs = tokenizer.apply_chat_template( - messages, - add_generation_prompt=True, - return_tensors="pt", - return_dict=True, -).to(model.device) - -outputs = model.generate( - **inputs, - max_new_tokens=200, - temperature=0.7 -) - -print(tokenizer.decode(outputs[0])) -``` - -## Chat template and tool calling - -OpenAI gpt-oss models use the [harmony response format](https://cookbook.openai.com/article/harmony) for structuring messages, including reasoning and tool calls. - -To construct prompts you can use the built-in chat template of Transformers. Alternatively, you can install and use the [openai-harmony library](https://github.com/openai/harmony) for more control. - -To use the chat template: - -```py -from transformers import AutoModelForCausalLM, AutoTokenizer - -model_name = "openai/gpt-oss-20b" - -tokenizer = AutoTokenizer.from_pretrained(model_name) -model = AutoModelForCausalLM.from_pretrained( - model_name, - device_map="auto", - torch_dtype="auto", -) - -messages = [ - {"role": "system", "content": "Always respond in riddles"}, - {"role": "user", "content": "What is the weather like in Madrid?"}, -] - -inputs = tokenizer.apply_chat_template( - messages, - add_generation_prompt=True, - return_tensors="pt", - return_dict=True, -).to(model.device) - -generated = model.generate(**inputs, max_new_tokens=100) -print(tokenizer.decode(generated[0][inputs["input_ids"].shape[-1] :])) -``` - -To integrate the [`openai-harmony`](https://github.com/openai/harmony) library to prepare prompts and parse responses, first install it like this: - -```bash -pip install openai-harmony -``` - -Here’s an example of how to use the library to build your prompts and encode them to tokens: - -```py -import json -from openai_harmony import ( - HarmonyEncodingName, - load_harmony_encoding, - Conversation, - Message, - Role, - SystemContent, - DeveloperContent -) -from transformers import AutoModelForCausalLM, AutoTokenizer - -encoding = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS) - -# Build conversation -convo = Conversation.from_messages([ - Message.from_role_and_content(Role.SYSTEM, SystemContent.new()), - Message.from_role_and_content( - Role.DEVELOPER, - DeveloperContent.new().with_instructions("Always respond in riddles") - ), - Message.from_role_and_content(Role.USER, "What is the weather like in SF?") -]) - -# Render prompt -prefill_ids = encoding.render_conversation_for_completion(convo, Role.ASSISTANT) -stop_token_ids = encoding.stop_tokens_for_assistant_actions() - -# Load model -model_name = "openai/gpt-oss-20b" -tokenizer = AutoTokenizer.from_pretrained(model_name) -model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto") - -# Generate -outputs = model.generate( - input_ids=[prefill_ids], - max_new_tokens=128, - eos_token_id=stop_token_ids -) - -# Parse completion tokens -completion_ids = outputs[0][len(prefill_ids):] -entries = encoding.parse_messages_from_completion_tokens(completion_ids, Role.ASSISTANT) - -for message in entries: - print(json.dumps(message.to_dict(), indent=2)) -``` - -Note that the `Developer` role in Harmony maps to the `system` prompt in the chat template. - -## Multi-GPU & distributed inference - -The large gpt-oss-120b fits on a single H100 GPU when using MXFP4. If you want to run it on multiple GPUs, you can: - -- Use `tp_plan="auto"` for automatic placement and tensor parallelism -- Launch with `accelerate launch or torchrun` for distributed setups -- Leverage Expert Parallelism -- Use specialised Flash attention kernels for faster inference - -```py -from transformers import AutoModelForCausalLM, AutoTokenizer -from transformers.distributed import DistributedConfig -import torch - -model_path = "openai/gpt-oss-120b" -tokenizer = AutoTokenizer.from_pretrained(model_path, padding_side="left") - -device_map = { - # Enable Expert Parallelism - "distributed_config": DistributedConfig(enable_expert_parallel=1), - # Enable Tensor Parallelism - "tp_plan": "auto", -} - -model = AutoModelForCausalLM.from_pretrained( - model_path, - torch_dtype="auto", - attn_implementation="kernels-community/vllm-flash-attn3", - **device_map, -) - -messages = [ - {"role": "user", "content": "Explain how expert parallelism works in large language models."} -] - -inputs = tokenizer.apply_chat_template( - messages, - add_generation_prompt=True, - return_tensors="pt", - return_dict=True, -).to(model.device) - -outputs = model.generate(**inputs, max_new_tokens=1000) - -# Decode and print -response = tokenizer.decode(outputs[0]) -print("Model response:", response.split("<|channel|>final<|message|>")[-1].strip()) -``` - -You can then run this on a node with four GPUs via - -```bash -torchrun --nproc_per_node=4 generate.py -``` diff --git a/articles/how_to_work_with_large_language_models.md b/articles/how_to_work_with_large_language_models.md deleted file mode 100644 index cf6b48e1be..0000000000 --- a/articles/how_to_work_with_large_language_models.md +++ /dev/null @@ -1,168 +0,0 @@ -# How to work with large language models - -## How large language models work - -[Large language models][Large language models Blog Post] are functions that map text to text. Given an input string of text, a large language model predicts the text that should come next. - -The magic of large language models is that by being trained to minimize this prediction error over vast quantities of text, the models end up learning concepts useful for these predictions. For example, they learn: - -- how to spell -- how grammar works -- how to paraphrase -- how to answer questions -- how to hold a conversation -- how to write in many languages -- how to code -- etc. - -They do this by “reading” a large amount of existing text and learning how words tend to appear in context with other words, and uses what it has learned to predict the next most likely word that might appear in response to a user request, and each subsequent word after that. - -GPT-3 and GPT-4 power [many software products][OpenAI Customer Stories], including productivity apps, education apps, games, and more. - -## How to control a large language model - -Of all the inputs to a large language model, by far the most influential is the text prompt. - -Large language models can be prompted to produce output in a few ways: - -- **Instruction**: Tell the model what you want -- **Completion**: Induce the model to complete the beginning of what you want -- **Scenario**: Give the model a situation to play out -- **Demonstration**: Show the model what you want, with either: - - A few examples in the prompt - - Many hundreds or thousands of examples in a fine-tuning training dataset - -An example of each is shown below. - -### Instruction prompts - -Write your instruction at the top of the prompt (or at the bottom, or both), and the model will do its best to follow the instruction and then stop. Instructions can be detailed, so don't be afraid to write a paragraph explicitly detailing the output you want, just stay aware of how many [tokens](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them) the model can process. - -Example instruction prompt: - -```text -Extract the name of the author from the quotation below. - -“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.” -― Ted Chiang, Exhalation -``` - -Output: - -```text -Ted Chiang -``` - -### Completion prompt example - -Completion-style prompts take advantage of how large language models try to write text they think is most likely to come next. To steer the model, try beginning a pattern or sentence that will be completed by the output you want to see. Relative to direct instructions, this mode of steering large language models can take more care and experimentation. In addition, the models won't necessarily know where to stop, so you will often need stop sequences or post-processing to cut off text generated beyond the desired output. - -Example completion prompt: - -```text -“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.” -― Ted Chiang, Exhalation - -The author of this quote is -``` - -Output: - -```text - Ted Chiang -``` - -### Scenario prompt example - -Giving the model a scenario to follow or role to play out can be helpful for complex queries or when seeking imaginative responses. When using a hypothetical prompt, you set up a situation, problem, or story, and then ask the model to respond as if it were a character in that scenario or an expert on the topic. - -Example scenario prompt: - -```text -Your role is to extract the name of the author from any given text - -“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.” -― Ted Chiang, Exhalation -``` - -Output: - -```text - Ted Chiang -``` - -### Demonstration prompt example (few-shot learning) - -Similar to completion-style prompts, demonstrations can show the model what you want it to do. This approach is sometimes called few-shot learning, as the model learns from a few examples provided in the prompt. - -Example demonstration prompt: - -```text -Quote: -“When the reasoning mind is forced to confront the impossible again and again, it has no choice but to adapt.” -― N.K. Jemisin, The Fifth Season -Author: N.K. Jemisin - -Quote: -“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.” -― Ted Chiang, Exhalation -Author: -``` - -Output: - -```text - Ted Chiang -``` - -### Fine-tuned prompt example - -With enough training examples, you can [fine-tune][Fine Tuning Docs] a custom model. In this case, instructions become unnecessary, as the model can learn the task from the training data provided. However, it can be helpful to include separator sequences (e.g., `->` or `###` or any string that doesn't commonly appear in your inputs) to tell the model when the prompt has ended and the output should begin. Without separator sequences, there is a risk that the model continues elaborating on the input text rather than starting on the answer you want to see. - -Example fine-tuned prompt (for a model that has been custom trained on similar prompt-completion pairs): - -```text -“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.” -― Ted Chiang, Exhalation - -### - - -``` - -Output: - -```text - Ted Chiang -``` - -## Code Capabilities - -Large language models aren't only great at text - they can be great at code too. OpenAI's [GPT-4][GPT-4 and GPT-4 Turbo] model is a prime example. - -GPT-4 powers [numerous innovative products][OpenAI Customer Stories], including: - -- [GitHub Copilot] (autocompletes code in Visual Studio and other IDEs) -- [Replit](https://replit.com/) (can complete, explain, edit and generate code) -- [Cursor](https://cursor.sh/) (build software faster in an editor designed for pair-programming with AI) - -GPT-4 is more advanced than previous models like `gpt-3.5-turbo-instruct`. But, to get the best out of GPT-4 for coding tasks, it's still important to give clear and specific instructions. As a result, designing good prompts can take more care. - -### More prompt advice - -For more prompt examples, visit [OpenAI Examples][OpenAI Examples]. - -In general, the input prompt is the best lever for improving model outputs. You can try tricks like: - -- **Be more specific** E.g., if you want the output to be a comma separated list, ask it to return a comma separated list. If you want it to say "I don't know" when it doesn't know the answer, tell it 'Say "I don't know" if you do not know the answer.' The more specific your instructions, the better the model can respond. -- **Provide Context**: Help the model understand the bigger picture of your request. This could be background information, examples/demonstrations of what you want or explaining the purpose of your task. -- **Ask the model to answer as if it was an expert.** Explicitly asking the model to produce high quality output or output as if it was written by an expert can induce the model to give higher quality answers that it thinks an expert would write. Phrases like "Explain in detail" or "Describe step-by-step" can be effective. -- **Prompt the model to write down the series of steps explaining its reasoning.** If understanding the 'why' behind an answer is important, prompt the model to include its reasoning. This can be done by simply adding a line like "[Let's think step by step](https://arxiv.org/abs/2205.11916)" before each answer. - -[Fine Tuning Docs]: https://platform.openai.com/docs/guides/fine-tuning -[OpenAI Customer Stories]: https://openai.com/customer-stories -[Large language models Blog Post]: https://openai.com/research/better-language-models -[GitHub Copilot]: https://github.com/features/copilot/ -[GPT-4 and GPT-4 Turbo]: https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo -[GPT3 Apps Blog Post]: https://openai.com/blog/gpt-3-apps/ -[OpenAI Examples]: https://platform.openai.com/examples diff --git a/articles/openai-cookbook-llms-101.md b/articles/openai-cookbook-llms-101.md new file mode 100644 index 0000000000..f306effc2f --- /dev/null +++ b/articles/openai-cookbook-llms-101.md @@ -0,0 +1,184 @@ +--- +title: "LLMs 101: A Practical Introduction" +description: "A hands-on, code-first introduction to large language models for Cookbook readers." +last_updated: "2025-08-24" +--- + +# LLMs 101: A Practical Introduction + +> **Who this is for.** Developers who want a fast, working understanding of large language models and the knobs that matter in real apps. + +## At a glance + +``` +Text prompt + ↓ (tokenization) +Tokens → Embeddings → [Transformer layers × N] → Next‑token probabilities + ↓ ↓ +Detokenization Sampling (temperature/top_p) → Output text +``` + +- **LLMs** are neural networks (usually **transformers**) trained on lots of text to predict the next token. +- **Tokenization** splits text into subword units; **embeddings** map tokens to vectors; transformer layers build context‑aware representations. +- Generation repeats next‑token sampling until a stop condition (length or stop sequences) is met. + +--- + +## Quick start: generate text + +### Python + +```python +from openai import OpenAI + +client = OpenAI() +resp = client.responses.create( + model="gpt-4o", + instructions="You are a concise technical explainer.", + input="In one paragraph, explain what a token is in an LLM." +) +print(resp.output_text) +``` + +### JavaScript / TypeScript + +```js +import OpenAI from "openai"; +const client = new OpenAI(); + +const resp = await client.chat.completions.create({ + model: "gpt-4o", + messages: [ + { role: "system", content: "You are a concise technical explainer." }, + { role: "user", content: "In one paragraph, explain what a token is in an LLM." } + ] +}); +console.log(resp.choices[0].message.content); +``` + +> **Tip.** Model names evolve; check your Models list before shipping. Prefer streaming for chat‑like UIs (see below). + +--- + +## What can LLMs do? + +Despite the name, LLMs can be **multi‑modal** when models and inputs support it (text, code, sometimes images/audio). Core text tasks: + +- **Generate**: draft, rewrite, continue, or brainstorm. +- **Transform**: translate, rephrase, format, classify, extract. +- **Analyze**: summarize, compare, tag, or answer questions. +- **Tool use / agents**: call functions or APIs as part of a loop to act. + +These patterns compose into search, assistants, form‑fillers, data extraction, QA, and more. + +--- + +## How LLMs work (just enough to be dangerous) + +1. **Tokenization.** Input text → tokens (IDs). Whitespace and punctuation matter—“token‑budget math” is a real constraint. +2. **Embeddings.** Each token ID becomes a vector; positions are encoded so order matters. +3. **Transformer layers.** Self‑attention mixes information across positions so each token’s representation becomes **contextual** (richer than the raw embedding). +4. **Decoding.** The model outputs a probability distribution over the next token. +5. **Sampling.** Choose how “adventurous” generation is (see knobs below), append the token, and repeat until done. + +--- + +## The knobs you’ll touch most + +- **Temperature** *(0.0–2.0)* — Lower → more deterministic/boring; higher → more diverse/creative. +- **Top‑p (nucleus)** *(0–1)* — Sample only from the smallest set of tokens whose cumulative probability ≤ *p*. +- **Max output tokens** — Hard limit on output length; controls latency and cost. +- **System / instructions** — Up‑front role, constraints, and style to steer behavior. +- **Stop sequences** — Cleanly cut off output at known boundaries. +- **Streaming** — Receive tokens as they’re generated; improves perceived latency. + +**Practical defaults:** `temperature=0.2–0.7`, `top_p=1.0`, set a **max output** that fits your UI, and **stream** by default for chat UX. + +--- + +## Make context do the heavy lifting + +- **Context window.** Inputs + outputs share a finite token budget; plan prompts and retrieval to fit. +- **Ground with your data (RAG).** Retrieve relevant snippets and include them in the prompt to improve factuality. +- **Structured outputs.** Ask for JSON (and validate) when you need machine‑readable results. +- **Few‑shot examples.** Provide 1–3 compact exemplars to stabilize format and tone. + +--- + +## Minimal streaming example + +### Python + +```python +from openai import OpenAI +client = OpenAI() + +with client.responses.stream( + model="gpt-4o", + input="Stream a two-sentence explanation of context windows." +) as stream: + for event in stream: + if event.type == "response.output_text.delta": + print(event.delta, end="") +``` + +### JavaScript + +```js +import OpenAI from "openai"; +const client = new OpenAI(); + +const stream = await client.responses.stream({ + model: "gpt-4o", + input: "Stream a two-sentence explanation of context windows." +}); + +for await (const event of stream) { + if (event.type === "response.output_text.delta") { + process.stdout.write(event.delta); + } +} +``` + +--- + +## Limitations (design around these) + +- **Hallucinations.** Models can generate plausible but false statements. Ground with citations/RAG; validate critical outputs. +- **Recency.** Models don’t inherently know the latest facts; retrieve or provide current data. +- **Ambiguity.** Vague prompts → vague answers; specify domain, audience, length, and format. +- **Determinism.** Even at `temperature=0`, responses may vary across runs/envs. Don’t promise bit‑for‑bit reproducibility. +- **Cost & latency.** Longer prompts and bigger models are slower and costlier; iterate toward the smallest model that meets quality. + +--- + +## Common gotchas + +- **Characters ≠ tokens.** Budget both input and output to avoid truncation. +- **Over‑prompting.** Prefer simple, testable instructions; add examples sparingly. +- **Leaky formats.** If you need JSON, enforce it (schema + validators) and add a repair step. +- **One prompt for everything.** Separate prompts per task/endpoint; keep them versioned and testable. +- **Skipping evaluation.** Keep a tiny dataset of real tasks; score changes whenever you tweak prompts, models, or retrieval. + +--- + +## Glossary + +- **Token** — Small unit of text (≈ subword) used by models. +- **Embedding** — Vector representation of a token or text span. +- **Context window** — Max tokens the model can attend to at once (prompt + output). +- **Temperature / top‑p** — Randomness controls during sampling. +- **System / instructions** — Up‑front guidance that shapes responses. +- **RAG** — Retrieval‑Augmented Generation; retrieve data and include it in the prompt. + +--- + +## Where to go next + +- Prompt patterns for **structured outputs** +- **Retrieval‑augmented generation (RAG)** basics +- **Evaluating** LLM quality (offline + online) +- **Streaming UX** patterns and backpressure handling +- **Safety** and policy‑aware prompting + +> Adapted from a shorter draft and expanded with code-first guidance. diff --git a/authors.yaml b/authors.yaml index cd407f6c69..0faf6d884b 100644 --- a/authors.yaml +++ b/authors.yaml @@ -3,11 +3,6 @@ # You can optionally customize how your information shows up cookbook.openai.com over here. # If your information is not present here, it will be pulled from your GitHub profile. -daveleo-openai: - name: "Dave Leo" - website: "https://www.linkedin.com/in/davidanthonyleo/" - avatar: "https://media.licdn.com/dms/image/v2/C5603AQF2Kg-D7XJKNw/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1612654752234?e=1761782400&v=beta&t=RkO9jCbJrY6Ox9YRbMA6HAAZhxfYJV1OsZeIT3YatBM" - jonlim-openai: name: "Jonathan Lim" website: "https://www.linkedin.com/in/jonlmr" @@ -493,10 +488,7 @@ heejingithub: website: "https://www.linkedin.com/in/heejc/" avatar: "https://avatars.githubusercontent.com/u/169293861" - -himadri: - name: "Himadri Acharya" - website: "https://www.linkedin.com/in/himadri-acharya-086ba261/" - avatar: "https://avatars.githubusercontent.com/u/14100684?v=4" - - \ No newline at end of file +paytonison: + name: "Payton Ison" + website: "https://linkedin.com/in/paytonison" + avatar: "https://avatars.githubusercontent.com/u/148833579" diff --git a/examples/agents_sdk/multi-agent-portfolio-collaboration/multi_agent_portfolio_collaboration.ipynb b/examples/agents_sdk/multi-agent-portfolio-collaboration/multi_agent_portfolio_collaboration.ipynb index aa505230ab..3c7e0423af 100644 --- a/examples/agents_sdk/multi-agent-portfolio-collaboration/multi_agent_portfolio_collaboration.ipynb +++ b/examples/agents_sdk/multi-agent-portfolio-collaboration/multi_agent_portfolio_collaboration.ipynb @@ -497,7 +497,7 @@ "- **Function tools:** Register any Python function as a tool, with automatic schema and validation.\n", "- **Tracing:** Visualize, debug, and monitor every step of your workflow for full transparency.\n", "\n", - "A combination of well-designed tools, thoughtful orchestration, and careful model selection is crucial for building effective agent systems. In this example, we use the GPT-4.1 family of models for their strong analytical and tool-use capabilities ([see the GPT-4.1 Prompting Guide](https://cookbook.openai.com/examples/gpt4-1_prompting_guide)). For deeper architectural best practices, see the included [A Practical Guide to Building Agents (PDF)](https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf). By bringing these elements together, you get a system that is robust, scalable, and easy to debug or extend.\n", + "A combination of well-designed tools, thoughtful orchestration, and careful model selection is crucial for building effective agent systems. In this example, we use the GPT-4.1 family of models for their strong analytical and tool-use capabilities ([see the GPT-4.1 Prompting Guide](https://cookbook.openai.com/examples/prompting/gpt4-1_prompting_guide)). For deeper architectural best practices, see the included [A Practical Guide to Building Agents (PDF)](https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf). By bringing these elements together, you get a system that is robust, scalable, and easy to debug or extend.\n", "\n", "Please try out the sample with your own investment questions, and please share any feedback! Happy building.\n", "\n", @@ -512,7 +512,7 @@ "\n", "- [MCP Spec](https://spec.modelcontextprotocol.io/specification/2024-11-05/architecture/)\n", "- [OpenAI Cookbook](https://github.com/openai/openai-cookbook)\n", - "- ([GPT-4.1 Prompting Guide](https://cookbook.openai.com/examples/gpt4-1_prompting_guide))\n", + "- ([GPT-4.1 Prompting Guide](https://cookbook.openai.com/examples/prompting/gpt4-1_prompting_guide))\n", "- [A Practical Guide to Building Agents (PDF)](https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf)\n", "\n", "---" diff --git a/examples/agents_sdk/multi-agent-portfolio-collaboration/utils.py b/examples/agents_sdk/multi-agent-portfolio-collaboration/utils.py index 4d3af6c8b0..85f491df75 100644 --- a/examples/agents_sdk/multi-agent-portfolio-collaboration/utils.py +++ b/examples/agents_sdk/multi-agent-portfolio-collaboration/utils.py @@ -39,7 +39,7 @@ def outputs_dir() -> Path: # Prompt loader # --------------------------------------------------------------------------- -PROMPTS_DIR: Path = repo_path("prompts") +PROMPTS_DIR: Path = repo_path("prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts") def load_prompt(name: str, **subs) -> str: diff --git a/examples/codex/Autofix-github-actions.ipynb b/examples/codex/Autofix-github-actions.ipynb deleted file mode 100644 index b492626bb1..0000000000 --- a/examples/codex/Autofix-github-actions.ipynb +++ /dev/null @@ -1,223 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "e2884696", - "metadata": {}, - "source": [ - "# Autofix CI failures on GitHub with Codex-cli\n", - "\n", - "## Purpose of this cookbook\n", - "\n", - "This cookbook shows you how to embed the OpenAI Codex CLI into your CI/CD pipeline so that when your builds or tests fail, codex automatically generates & proposes fixes. The following is an example in a node project with CI running in GitHub Actions. \n", - "\n", - "## End to End Flow\n", - "\n", - "Below is the pipeline flow we’ll implement:\n", - "\n", - "![](images/ci-codex-workflow.png)" - ] - }, - { - "cell_type": "markdown", - "id": "f83ce964", - "metadata": {}, - "source": [ - "## Prerequisites\n", - "\n", - "- A GitHub Repo with Actions workflows\n", - "\n", - "- You’ll need to create `OPENAI_API_KEY` as an environment variable in GitHub settings under https://github.com/{org-name}/{repo-name}/settings/secrets/actions. You can also set this at org level(for sharing secrets across multiple repos) \n", - "\n", - "- Codex requires python as a prerequisite to use `codex login`\n", - "\n", - "- You’ll need to check the setting to enable actions to create PRs on your repo, and also in your organization:\n", - "\n", - "![](images/github-pr-settings.png)" - ] - }, - { - "cell_type": "markdown", - "id": "99f5bed1", - "metadata": {}, - "source": [ - "\n", - "## Step 3: Insert Codex in your CI pipeline\n", - "\n", - "The following YAML shows a GitHub action that auto triggers when CI fails, installs Codex, uses codex exec and then makes a PR on the failing branch with the fix. Replace \"CI\" with the name of the workflow you want to monitor. " - ] - }, - { - "cell_type": "markdown", - "id": "a9f9b368", - "metadata": {}, - "source": [ - "```yaml\n", - "\n", - "name: Codex Auto-Fix on Failure\n", - "\n", - "on:\n", - " workflow_run:\n", - " # Trigger this job after any run of the primary CI workflow completes\n", - " workflows: [\"CI\"]\n", - " types: [completed]\n", - "\n", - "permissions:\n", - " contents: write\n", - " pull-requests: write\n", - "\n", - "jobs:\n", - " auto-fix:\n", - " # Only run when the referenced workflow concluded with a failure\n", - " if: ${{ github.event.workflow_run.conclusion == 'failure' }}\n", - " runs-on: ubuntu-latest\n", - " env:\n", - " OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n", - " FAILED_WORKFLOW_NAME: ${{ github.event.workflow_run.name }}\n", - " FAILED_RUN_URL: ${{ github.event.workflow_run.html_url }}\n", - " FAILED_HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}\n", - " FAILED_HEAD_SHA: ${{ github.event.workflow_run.head_sha }}\n", - " steps:\n", - " - name: Check prerequisites\n", - " run: |\n", - " if [ -z \"$OPENAI_API_KEY\" ]; then\n", - " echo \"OPENAI_API_KEY secret is not set. Skipping auto-fix.\" >&2\n", - " exit 1\n", - " fi\n", - "\n", - " - name: Checkout failing ref\n", - " uses: actions/checkout@v4\n", - " with:\n", - " ref: ${{ env.FAILED_HEAD_SHA }}\n", - " fetch-depth: 0\n", - "\n", - " - name: Setup Node.js\n", - " uses: actions/setup-node@v4\n", - " with:\n", - " node-version: '20'\n", - " cache: 'npm'\n", - "\n", - " - name: Install dependencies\n", - " run: |\n", - " if [ -f package-lock.json ]; then npm ci; else npm i; fi\n", - "\n", - " - name: Prepare Codex prerequisites\n", - " shell: bash\n", - " run: |\n", - " # Ensure python3 exists for Codex' login helper\n", - " if ! command -v python3 >/dev/null 2>&1; then\n", - " sudo apt-get update\n", - " sudo apt-get install -y python3\n", - " fi\n", - "\n", - " # Ensure Codex config dir exists and is writable\n", - " mkdir -p \"$HOME/.codex\"\n", - " # (Optional) pin an explicit home for Codex config/logs\n", - " echo \"CODEX_HOME=$HOME/.codex\" >> $GITHUB_ENV\n", - "\n", - " - name: Install Codex CLI\n", - " run: npm i -g @openai/codex\n", - "\n", - " - name: Authenticate Codex (non-interactive)\n", - " env:\n", - " # if you set CODEX_HOME above, export it here too\n", - " CODEX_HOME: ${{ env.CODEX_HOME }}\n", - " OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n", - " run: codex login --api-key \"$OPENAI_API_KEY\"\n", - "\n", - " - name: Run Codex to fix CI failure\n", - " run: |\n", - " codex exec --full-auto --sandbox workspace-write \"You are working in a Node.js monorepo with Jest tests and GitHub Actions. Read the repository, run the test suite, identify the minimal change needed to make all tests pass, implement only that change, and stop. Do not refactor unrelated code or files. Keep changes small and surgical.\"\n", - "\n", - " - name: Verify tests\n", - " run: npm test --silent\n", - "\n", - " - name: Create pull request with fixes\n", - " if: success()\n", - " uses: peter-evans/create-pull-request@v6\n", - " with:\n", - " commit-message: \"fix(ci): auto-fix failing tests via Codex\"\n", - " branch: codex/auto-fix-${{ github.event.workflow_run.run_id }}\n", - " base: ${{ env.FAILED_HEAD_BRANCH }}\n", - " title: \"Auto-fix failing CI via Codex\"\n", - " body: |\n", - " Codex automatically generated this PR in response to a CI failure on workflow `${{ env.FAILED_WORKFLOW_NAME }}`.\n", - "\n", - " Failed run: ${{ env.FAILED_RUN_URL }}\n", - " Head branch: `${{ env.FAILED_HEAD_BRANCH }}`\n", - "\n", - " This PR contains minimal changes intended solely to make the CI pass.\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "8148024b", - "metadata": {}, - "source": [ - "## Step 4: Actions Workflow kicked off\n", - "\n", - "You can navigate to the Actions tab under Repo to view the failing jobs in your Actions workflow. \n", - "\n", - "\n", - "![](images/failing-workflow.png)\n" - ] - }, - { - "cell_type": "markdown", - "id": "64671aae", - "metadata": {}, - "source": [ - "The Codex workflow should be triggered upon completion of the failed workflow. \n", - "\n", - "\n", - "![](images/codex-workflow.png)\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "id": "d08a3ecc", - "metadata": {}, - "source": [ - "## Step 5: Codex generated PR for review\n", - "And after the Codex workflow completes execution, it should open a pull request from the feature branch codex/auto-fix. Check to see if everything looks good and then merge it.\n", - "\n", - "![](images/codex-pr.png)" - ] - }, - { - "cell_type": "markdown", - "id": "f4c1f3a0", - "metadata": {}, - "source": [ - "## Conclusion\n", - "\n", - "This automation seamlessly integrates OpenAI Codex CLI with GitHub Actions to automatically propose fixes for failing CI runs.\n", - "\n", - "By leveraging Codex, you can reduce manual intervention, accelerate code reviews, and keep your main branch healthy. The workflow ensures that test failures are addressed quickly and efficiently, letting developers focus on higher-value tasks. Explore more about codex-cli and its capabilities [here](https://github.com/openai/codex/)." - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.13.7" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/examples/codex/images/ci-codex-workflow.png b/examples/codex/images/ci-codex-workflow.png deleted file mode 100644 index 4be3dc4fb4..0000000000 Binary files a/examples/codex/images/ci-codex-workflow.png and /dev/null differ diff --git a/examples/codex/images/codex-pr.png b/examples/codex/images/codex-pr.png deleted file mode 100644 index f9ba75ca12..0000000000 Binary files a/examples/codex/images/codex-pr.png and /dev/null differ diff --git a/examples/codex/images/codex-workflow.png b/examples/codex/images/codex-workflow.png deleted file mode 100644 index ffa1b7f71f..0000000000 Binary files a/examples/codex/images/codex-workflow.png and /dev/null differ diff --git a/examples/codex/images/failing-workflow.png b/examples/codex/images/failing-workflow.png deleted file mode 100644 index 342f6811b3..0000000000 Binary files a/examples/codex/images/failing-workflow.png and /dev/null differ diff --git a/examples/codex/images/github-pr-settings.png b/examples/codex/images/github-pr-settings.png deleted file mode 100644 index 723e267179..0000000000 Binary files a/examples/codex/images/github-pr-settings.png and /dev/null differ diff --git a/examples/fine-tuned_qa/reinforcement_finetuning_healthbench.ipynb b/examples/fine-tuned_qa/reinforcement_finetuning_healthbench.ipynb index c0ca41bad1..496e3fee14 100644 --- a/examples/fine-tuned_qa/reinforcement_finetuning_healthbench.ipynb +++ b/examples/fine-tuned_qa/reinforcement_finetuning_healthbench.ipynb @@ -15,7 +15,7 @@ "\n", "### HealthBench\n", "\n", - "This cookbook evaluates and improves model performance on a focused subset of [HealthBench](https://openai.com/index/healthbench/), a benchmark suite for medical QA. It walks through how to configure the datasets, define evaluation rubrics, and fine-tune model behavior using reinforcement signals derived from custom graders.\n", + "This cookbook evaluates and improves model performance on a focused subset of [HealthBench](https://openai.com/index/healthbench/), a benchmark suite for medical QA. This guide walks through how to configure the datasets, define evaluation rubrics, and fine-tune model behavior using reinforcement signals derived from custom graders.\n", "\n", "HealthBench is a comprehensive evaluation benchmark developed to assess the performance of large language models on healthcare-related question answering. It spans multiple clinical domains and question types, emphasizing accuracy, safety, and factual grounding.\n", "\n", @@ -23,7 +23,7 @@ "\n", "The [openai/simple-evals](https://github.com/openai/simple-evals) repository is a lightweight framework for prototyping and running evaluation pipelines on OpenAI models. It’s designed to support both structured and unstructured inputs, flexible grader configurations, and integration with OpenAI's fine-tuning APIs.\n", "\n", - "We will use this framework to evaluate the performance of GPT-4.1 on a focused subset of HealthBench so we can perform some error analysis on where the model is making mistakes.\n" + "We will use this framework to evaluate the performance of GPT 4.1 on a focused subset of HealthBench so we can perform some error analysis on where the model is making mistakes.\n" ] }, { @@ -40,9 +40,9 @@ "pip install openai human-eval\n", "```\n", "\n", - "2. GPT-4.1 is one of the best performing models on [HealthBench hard](https://openai.com/index/healthbench/). For a more detailed breakdown of the results on HealthBench, check out the [healthbench_analysis](https://github.com/openai/simple-evals/blob/main/healthbench_scripts/healthbench_analysis.ipynb) notebook.\n", + "2. GPT 4.1 is one of the best performing models on [HealthBench hard](https://openai.com/index/healthbench/). For a more detailed breakdown of the results on HealthBench, checkout the [healthbench_analysis](https://github.com/openai/simple-evals/blob/main/healthbench_scripts/healthbench_analysis.ipynb) notebook.\n", "\n", - "Run the command below\n", + "Run the below command\n", "```bash\n", "python -m simple-evals.simple_evals --eval=healthbench_hard --model=gpt-4.1\n", "```\n", @@ -94,23 +94,31 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "id": "8db1b3e4", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], "source": [ "%pip install openai evals matplotlib tqdm rich --upgrade --quiet" ] }, { "cell_type": "code", - "execution_count": 29, + "execution_count": null, "id": "62e77894", "metadata": {}, "outputs": [], "source": [ "import json\n", - "from collections import Counter\n", + "from collections import Counter, defaultdict\n", "import time\n", "\n", "import matplotlib.pyplot as plt\n", @@ -124,7 +132,7 @@ "from rich.text import Text\n", "from rich.console import Console\n", "\n", - "import os\n", + "\n", "client = OpenAI()" ] }, @@ -135,16 +143,16 @@ "source": [ "## Create training and validation datasets\n", "\n", - "In this section, we will create a training and validation dataset from a subset of HealthBench Hard. For this example, we will use just 12 datapoints from this the dataset to keep the training time short for the tutorial.\n", + "In this section, we will create a training and validation dataset from a subset of HealthBench Hard. For this example, we will use the first 24 datapoints from the dataset with the shortest prompts to keep the training time short for the tutorial.\n", "\n", "The challenging rubric that even some of the best models struggle with on HealthBench Hard requires the models to seek additional context to reduce uncertainty. We will use this rubric for our fine-tuning job.\n", "\n", - "We also visualize the distribution of how well models performed without any RFT. The justification scores illustrate there is a diverse range of scores in our training and validation datasets but it's skewed towards not meeting the rubric (a score of 0 denotes not meeting the rubric).\n" + "We also visualize the distribution of how well models performed without any RFT. The explanation scores illustrate there is a diverse range of scores in our training and validation datasets but its skewed towards not meeting the rubric (a score of 0 denotes not meeting the rubric).\n" ] }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 3, "id": "4f02e651", "metadata": {}, "outputs": [], @@ -163,8 +171,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Counter(data['criteria_met']): Counter({False: 44, True: 9})\n", - "Counter(filtered_data['criteria_met']): Counter({False: 17, True: 6})\n" + "Counter(data['criteria_met']): Counter({False: 44, True: 9})\n" ] } ], @@ -176,7 +183,12 @@ "# From analysing the results, we can observe this is one of the rubrics that is particularly challenging for models in HealthBench Hard.\n", "CHALLENGING_RUBRIC = \"\"\"The response seeks additional context to reduce uncertainty, either by asking follow-up questions or by suggesting what additional information would be helpful. The response should seek the most informative or useful context.\"\"\"\n", "\n", - "data = []\n", + "data = {\n", + " 'prompts': [],\n", + " 'completions': [],\n", + " 'criteria_met': [],\n", + " 'explanations': []\n", + "}\n", "\n", "for example in results['metadata']['example_level_metadata']:\n", " rubric_items = [\n", @@ -186,33 +198,69 @@ " \n", " if rubric_items:\n", " item = rubric_items[0]\n", - " data.append(\n", - " {\n", - " 'criteria_met': item['criteria_met'],\n", - " 'explanation': item['explanation'],\n", - " 'prompt': example['prompt'],\n", - " 'completion': example['completion']\n", - " }\n", - " )\n", + " data['criteria_met'].append(item['criteria_met'])\n", + " data['explanations'].append(item['explanation'])\n", + " data['prompts'].append(example['prompt'])\n", + " data['completions'].append(example['completion'])\n", "\n", "# Few of the examples meet the criteria\n", - "print(\"Counter(data['criteria_met']):\", Counter([datapoint['criteria_met'] for datapoint in data]))\n", - "\n", - "# Only include examples that have been pre-filtered to make the RFT job simple to run and evaluate\n", - "filter_indices = set(\n", - " [0, 1, 2, 7, 8, 9, 10, 12, 15, 20, 21, 26, 27, 30, 35, 38, 39, 41, 44, 45, 47, 49, 50]\n", - ")\n", - "filtered_data = []\n", - "for i, datapoint in enumerate(data):\n", - " if i in filter_indices:\n", - " filtered_data.append(datapoint)\n", - "\n", - "print(\"Counter(filtered_data['criteria_met']):\", Counter([datapoint['criteria_met'] for datapoint in filtered_data]))" + "print(\"Counter(data['criteria_met']):\", Counter(data['criteria_met']))" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, + "id": "cf6fa9bf", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[0,\n", + " 1,\n", + " 2,\n", + " 5,\n", + " 7,\n", + " 9,\n", + " 10,\n", + " 12,\n", + " 15,\n", + " 18,\n", + " 20,\n", + " 21,\n", + " 25,\n", + " 26,\n", + " 30,\n", + " 32,\n", + " 33,\n", + " 35,\n", + " 38,\n", + " 39,\n", + " 44,\n", + " 45,\n", + " 49,\n", + " 50]" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Calculate total length of all strings in each prompt array\n", + "def total_prompt_length(prompt_array):\n", + " return sum(len(str(item['content'])) for item in prompt_array)\n", + "\n", + "# Find shortest prompts and their indices\n", + "sorted_prompts = sorted(data['prompts'], key=total_prompt_length)[:24]\n", + "shortest_indices = [i for i, prompt in enumerate(data['prompts']) if prompt in sorted_prompts]\n", + "shortest_indices" + ] + }, + { + "cell_type": "code", + "execution_count": 15, "id": "ed909ae9", "metadata": {}, "outputs": [ @@ -220,14 +268,14 @@ "name": "stderr", "output_type": "stream", "text": [ - "100%|██████████| 23/23 [00:35<00:00, 1.55s/it]\n" + "100%|██████████| 24/24 [00:34<00:00, 1.43s/it]\n" ] }, { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEyCAYAAAD+23eGAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjYsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvq6yFwwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAJFdJREFUeJzt3Qd0lGXWwPFLS2gh1IAIAQwlNAEV/CgWxAVpIqCCRkFgbZQAUZqQRECkrCILYhRWikoogqhgQUSKNBEISO8iUkykQwAV5jv32Z05eZOAmWSGaf/fOXPIvNOeCZP3ztPuzWWz2WwCAMD/5Lb/AACAIjAAACwIDAAACwIDAMCCwAAAsCAwAAAsCAwAAAsCAwDAgsAAALAgMAAALAgMN3D16lWJjY2VSpUqSYECBSQiIkJGjhwpZBEB4M/yeroB3mzs2LGSkJAgM2fOlJo1a8rGjRulW7duEhoaKtHR0Z5uHgC4RS6S6F1fmzZtpHTp0vL+++87jnXs2NH0Hj766COPtg0A3IWhpBto1KiRLFu2TPbu3Wuub926VVavXi0tW7b0dNMAwG0YSrqBwYMHy7lz5yQyMlLy5Mlj5hxGjRolUVFRnm4aALgNgeEG5s2bJ7NmzZLExEQzx7Blyxbp16+flC1bVrp27erp5gGAWzDHcAPly5c3vYZevXo5jr322mtmfmH37t0ebRsAuAtzDDeQmpoquXNbf0U6pHTt2jWPtQkA3I2hpBto27atmVMIDw83Q0lJSUkyfvx46d69u6ebBgBuw1DSDZw/f95scFu4cKEkJyebuYUnnnhC4uLiJCgoyNPNAwC3IDAAACyYYwAAWBAYAAAWBAYAgP+sStJlo8eOHZOQkBDJlSuXp5sDAB6hU8W6WEYXyKRfYh9wgUGDgm5CAwCIHDlyRMqVKxfYgUF7CvZfRpEiRTzdHADwCM3ppl+S7efEgA4M9uEjDQoEBgCBLpeLhtSZfAYAWBAYAAAWBAYAgAWBAQDgPYFBK6JpkrpKlSqZOsoREREycuRIsyYXAOAZHl2VNHbsWElISJCZM2eatNYbN26Ubt26SWhoqERHR3uyaQAQsDwaGNauXSvt2rWT1q1bm+sVK1aU2bNny4YNGzzZLAAIaB4NDI0aNZIpU6bI3r17pWrVqrJ161ZZvXq1KYaTmStXrphL2k0d9tQYVFUDEKiuufj859HAoPWU9eQeGRlpSmbqnINWTIuKisr0/qNHj5bhw4dnOJ6SkiKXL192+vXnH/hvYHGHRyPYcAfg5tA8SX4TGObNmyezZs2SxMREM8ewZcsW6devn0kE1bVr1wz3HzJkiMTExGTYBl6qVKls7Xy+cCyPuEtYWAm3PTcApJU/f37xm8AwYMAA02vo3LmzuV67dm05fPiw6RlkFhiCg4PNJT3NJpitjIJuzMjqigyHAOCJ841Hz16pqakZ3pAOKTFfAACe49EeQ9u2bc2cQnh4uBlKSkpKMhPP3bt392SzACCgeTQwTJo0yWxw69mzpyQnJ5u5heeff17i4uI82SwACGgeDQyaO3zChAnmAgDwDsyQAgAsCAwAAAsCAwDAgsAAALAgMAAALAgMAAALAgMAwILAAACwIDAAACwIDAAACwIDAMCCwAAAsCAwAAAsCAwAAAsCAwDAgsAAALAgMAAALAgMAAALAgMAwILAAACwIDAAACwIDAAACwIDAMCCwAAAsCAwAAByFhi+/vprWb16teP65MmTpW7duvLkk0/K6dOnnX06AICvB4YBAwbIuXPnzM/btm2Tl156SVq1aiWHDh2SmJgYd7QRAHAT5XX2ARoAatSoYX5esGCBtGnTRl5//XXZvHmzCRAAgADrMQQFBUlqaqr5+dtvv5XmzZubn4sXL+7oSQAAAqjH0KRJEzNk1LhxY9mwYYPMnTvXHN+7d6+UK1fOHW0EAHhzj+Htt9+WvHnzyvz58yUhIUFuvfVWc/yrr76Shx56yB1tBAB4c48hPDxcFi9enOH4W2+95ao2AQB8bR/DgQMHZNiwYfLEE09IcnKyo8ewY8cOV7cPAODtgWHlypVSu3Zt+eGHH+STTz6RCxcumONbt26V+Ph4d7QRAODNgWHw4MHy2muvydKlS80KJbsHHnhA1q9f7+r2AQC8PTDoprb27dtnOB4WFia///67q9oFAPCVwFC0aFE5fvx4huNJSUmOFUoAgAAKDJ07d5ZBgwbJiRMnJFeuXHLt2jVZs2aNvPzyy9KlSxf3tBIA4L2BQdNfREZGSvny5c3Es6bHuPfee6VRo0ZmpRIAIMD2MeiE89SpUyU2Nla2b99ugkO9evWkSpUq7mkhAMC7A0PajW56AQAEYGBwJp32+PHjc9IeAIAvBAZdcZQVOhkNAAiAwLB8+XK3NeDo0aNmlZOm1NB03pUrV5bp06fLXXfd5bbXBAC4YY5BHTlyxPyrK5SyQ0uBavrupk2bmsBQqlQp2bdvnxQrViwnzQIA3MzA8Ndff8nw4cNl4sSJjjxJhQsXlj59+phcSfny5cvyc40dO9YEFe0h2FWqVMnZJgEAPBkYNABo8rxx48ZJw4YNzbF169bJq6++KidPnjQ1GrLq888/lxYtWshjjz1mkvPpzumePXvKs88+m+n9r1y5Yi529opxuslOL06z2cRdstUeAPCC800um825s2NoaKjMmTNHWrZsaTn+5ZdfmjTcZ8+ezfJz5c+f37HqSYPDjz/+KH379pV3331XunbtmuH+Gny0t5KeVo8LCQkRZ80/4L5SpI9GFHHbcwNAWufPn5eqVaua82+RIkVufmDQZHn67b569eqW47t27TI7oFNSUpzaLKeTzGvXrnUci46ONgFCeyFZ6THoUJTOVWTnlzFuy0lxl4F1S7jtuQEgLT0X6tysqwKD00NJvXv3lpEjR5p5geDgYHNMT9ajRo0ytznjlltuMSk10tKAs2DBgkzvr69nf820cufObS5Oc+Py2my1BwC84HzjdGDQPQ3Lli2TcuXKSZ06dRxFev744w9p1qyZdOjQwXFfnYu4EV2RtGfPngzDQhUqVHC2WQAATwUGTbvdsWNHy7HsLlft37+/Sb6nifkef/xx2bBhg0yZMsVcAAA+EhjSLi3Nqfr168vChQtlyJAhMmLECLNUdcKECRIVFeWy1wAA3MQNbq7Qpk0bcwEA+Ghg0L0KcXFxJk1GcnJyhvWzp06dcmX7AADeHhiefvpp2b9/v/To0UNKly5N4jwACPTA8P3338vq1asdK5IAAP7F6cWvWtbz0qVL7mkNAMD3AsM777wjQ4cONbufdb5Bd9ylvQAAAnAfgwaABx54wHJcM2vofMPVq1dd2T4AgLcHBt1joKm1ExMTmXwGAD/kdGDYvn27SYtRrVo197QIAOBbcwyaDdVeuQ0A4H+yVahHayYMGDBAateunaFi2+233+7K9gEAvD0wdOrUyfzbvXt3xzGdZ2DyGQACNDAcOnTIPS0BAPhmYKBWAgD4t2xnV925c6f88ssvpkBPWg8//LAr2gUA8JXAcPDgQWnfvr1s27bNMbeg7PsZmGMAgABbrqorkrSgjqbcLliwoOzYsUNWrVpllrGuWLHCPa0EAHhvj2HdunXy3XffScmSJU0Bar00adJERo8eLdHR0WbzGwAggHoMOlQUEhJiftbgcOzYMcek9J49e1zfQgCAd/cYatWqJVu3bjXDSXfffbeMGzdOgoKCZMqUKXLbbbe5p5UAAO8NDMOGDZOLFy+an0eMGGHqNd9zzz1SokQJmTt3rjvaCADw5sDQokULx8+VK1eW3bt3mzrPxYoVI9MqAATiHENKSkqGY8WLFzdBQZewAmPGjDGfh379+nm6KQhwfBZvUmDQxHlffPFFhuNvvPGGNGjQIJvNgL/48ccf5b333iOZIjyOz+JNDAwxMTHSsWNHefHFF03t56NHj0qzZs3MJLQW70HgunDhginkNHXqVDO0CHgKn8WbHBgGDhxo9jJ8//33JhLrJTg4WH766SezIxqBq1evXtK6dWt58MEHPd0UBDg+ix7IlaSTzrpsdcGCBY5U3GXKlMlhU+DL5syZI5s3bzbdd8CT+Cx6oMewZs0a00vYt2+f6SUkJCSY4j0aHE6fPu2CJsHXaEU/TZUya9YsyZ8/v6ebgwDGZ9E1ctnsWfCySIeN+vfvLyNHjnRUbztw4IA89dRT5j/l119/lZvl3LlzEhoaKmfPnpUiRYo4/fgxSb+LuwyuV1ICxaeffmqGEfPkyWPZIa+rQTRlypUrVyy3Ae4SqJ/Fczk8F+Z4KOmbb76R++67z3IsIiLC9CRGjRqV4wbB9+jig/RLlbt16yaRkZEyaNAgv/xDhHfis+gaTgeG9EHBTqNxbGysK9oEH6O5s3TOKa1ChQqZ3fDpjwPuxGfxJs8xtGrVynRT0m4cOXPmjOP6yZMnpUaNGi5qFgDA6+cYtAt2/PhxCQsLM9d1HGvLli2OxHm//fablC1b9qYW6mGOAQDE5XMMWe4xpI8fTs5ZAwD8dbkqAMC/ZTkw6HKv9NlTyaYKAAG8KkmHjp555hmzj0FdvnxZXnjhBTPjr3R9MAAggAJD165dLdd1Q1t6Xbp0cU2rAADeHximT5/u3pYAALwCk88AAAsCAwDAgsAAALAgMAAAnA8Md9xxh6PWwogRIyQ1NTUrDwMA+Gtg2LVrl1y8eNH8PHz4cFNP1dU0KZ9umOvXr5/LnxsA4OLlqnXr1jU5zZs0aWI2ur3xxhtSuHDhTO8bFxcnztISfO+9956pDAcA8IHAMGPGDImPj5fFixebb/VfffWV5M2b8aF6m7OBQXsfUVFRMnXqVHnttdeceiwAwEOBoVq1aqbAtr0gz7Jlyxzpt3OqV69e0rp1a3nwwQf/NjBo2o20qTc01ay6du2auTjNjRlis9UeAPCC801eTzZAg83mzZvNUFJWjB492sxxpJeSkmJyNzmr8KX/BhZ3SE6+eXUp3G3+Aff9ntSjETnPH4/AwGcxc+fPnxePBgZ14MABmTBhgpmUVlq5rW/fvqb2c1YdOXLEPGbp0qWSP3/+LD1myJAhEhMTY+kxlC9fXkqVKpWt4hQXjrmv/mtYWAnxF+78Pfnb7wqB+VlMSEgw86Q///yzuV6zZk0ZNmyYtGzZUm6GrJ5D3RYYlixZIg8//LCZkG7cuLE5tmbNGvOLWLRokfzjH//I0vNs2rRJkpOTzVJYO63+tmrVKnn77bfNkFH6wt2a2dWe3TUtHd7Si9PcmDY8W+3xVm5Or+5XvysE5GcxPDzcrKysUqWKWaAzc+ZMad++vSQlJZlzo7u5+m/I6cAwePBg6d+/v/klpD8+aNCgLAeGZs2aybZt2yzHdOVTZGSkeZ70QQEAvFXbtm0t10eNGmV6EevXr78pgcHVnA4MOnw0b968DMe7d+9uhpeyKiQkRGrVqmU5prUdSpQokeE4APiKq1evyscff2z2fjVs2FB8kdOBQcfzt2zZYrpMaekxV61UAgBfs23bNhMIdCGM7vNauHChmX8NiMDw7LPPynPPPScHDx6URo0aOeYYxo4da5kYzo4VK1bk6PEA4CnVqlUzX5DPnj0r8+fPN8XNVq5c6ZPBwenAEBsba4aB3nzzTbNKSJUtW1ZeffVViY6OdkcbAcDrBQUFSeXKlc3Pd955p1mG/+9//9usVvL7wKC7m3XyWS/2tbMaKAAA1j1faTfk+pJs7WOwIyAAgJjRE92zoMtW9QtzYmKiGRrX5f0BFxgAAGL2ZHXp0kWOHz8uoaGhJiGoBoWsLt/3NgQGAMih999/X/wJW04BANkPDH/++afZsbxv3z5nHgYA8NfAkC9fPvnpp5/c1xoAgO8NJT311FN+N54GAMjB5PNff/0l06ZNk2+//dZs4tD8RmmNHz/e2acEAPhyYNi+fbsjVfbevXszbH4DAARYYFi+fLl7WgIA8O3lqvv37zcbOC5dumSua3EKAEAABoaTJ0+aJatVq1aVVq1amZ1+qkePHvLSSy+5o40AAG8ODJo8T5et/vLLL1KwYEHH8U6dOsnXX3/t6vYBALx9juGbb74xQ0jlypWzHNfCPYcPH3Zl2wAAvtBj0HJ1aXsKdqdOnZLg4GBXtQsA4CuB4Z577pEPPvjAskRV846PGzdOmjZt6ur2AQC8fShJA4BOPm/cuFH++OMPGThwoOzYscP0GLTEJwAgwHoMtWrVMhvbmjRpIu3atTNDSx06dJCkpCSJiIhwTysBAN5dj0ELUQwdOtT1rQEALzQm6Xe3PffgeiXFLwLD6dOnTSK9Xbt2mes1atSQbt26SfHixV3dvoA0evRo+eSTT2T37t1SoEABadSokYwdO1aqVavm6aYBCABODyWtWrVKKlasKBMnTjQBQi/6c6VKlcxtyLmVK1dKr169ZP369bJ06VJTB6N58+Zm2A4AvK7HoCcs3cyWkJAgefLkMceuXr0qPXv2NLdt27bNHe0MKOk3Cs6YMUPCwsJk06ZNcu+993qsXQACQ+7s5EjS1Bf2oKD055iYGHMbXO/s2bPmX4bqAHhlYNCU2/a5hbT0WJ06dVzVLvyP7hHp16+fNG7c2KwIAwCvGEpKW84zOjpa+vbta3oH//d//2eO6Vj45MmTZcyYMe5raYDS4TmtgbF69WpPNwVAgMhSYKhbt67Z4Zw2tbZubEvvySefNPMPcI3evXvL4sWLzaR++txUAODRwHDo0CG3NQAZaQDu06ePLFy4UFasWGFWfAGAVwWGChUquL8lsAwfJSYmymeffSYhISFy4sQJx8ZC3dcAAF63we3YsWNmzDs5OdlMjqalcxDIGV0KrO6//37L8enTp8szzzzjoVYBCBROBwZdU//8889LUFCQlChRwsw92OnPBIaco0wqAJ8KDLGxsRIXFydDhgyR3LmzXTIaAOClnD6zp6amSufOnQkKAOCnnD679+jRQz7++GP3tAYA4HtDSZr5s02bNiafT+3atSVfvnyW28ePH+/K9gEAfCEwLFmyxJECOv3kMwAgwALDm2++KdOmTWPZJAD4KafnGIKDg01CNwCAf3I6MGgCvUmTJrmnNQAA3xtK2rBhg3z33XcmuVvNmjUzTD5rSUoAQAAFhqJFi0qHDh3c0xoAgO8FBs3XAwDwXx7dvqxLX+vXr28yiGpN40ceeUT27NnjySYBQMBzusegtQFutF/h4MGDWX6ulStXmhTTGhz++usveeWVV6R58+ayc+dOKVSokLNNAwB4IjBo/eG0/vzzT0lKSjI7oQcMGODUc+lj0mdu1Z7Dpk2b5N5773W2aQAATwQGXa6aGa35vHHjxhw15uzZs+bf4sWLZ3r7lStXzMXu3Llz5l+tCZG+LkSWuDG9dbba463cnAbcr35X8M/Pos27zxWu/hvKVqGezLRs2dKk4s7u5LS+Me2N6Oa5WrVqXXdOYvjw4RmOp6SkyOXLl51+zcKX/htY3CE5+Wqmx+cfcN9rqkcjirj8Od35e7rR7wrwls9iYQ+cK5xx/vx58crAMH/+/Ot+088KnWvYvn27qQx3PRp4YmJiLD2G8uXLS6lSpaRIEedPiBeO5RF3CQsrcdNf80avmxPubvPu3TvkjTfekM2bN8vx48dlwYIFZiGCt1q1apVPtdefeOrv54IHzhXOyJ8/v3g0MNSrV88y+azVxrQmsX5rf+edd7LViN69e5sNc/oHV65cuRum49BLelobIlv1IdyY9O+67XFzokG31Mlwc5svXbokdevWNSnddY9Mtv8/bxJfa69f8dTfTy4PnCtu8nPkKDCk/2akDdJv7FqfODIy0qnn0qDSp08fWbhwoaxYscKseELg0WFIvfgKX2sv4PbAEB8fL66iw0eJiYny2Wefmb0M2vNQoaGhUqBAAZe9DgAg6zza/01ISDArkbS3ccsttzguc+fO9WSzACCgZbnHoENGf1eIR2/XjWrODCUBAHw0MOg8wPWsW7dOJk6cyHp0AAikwNCuXbsMxzSv0eDBg2XRokUSFRUlI0aMcHX7AAA3Wbb2MRw7dsxMQs+cOVNatGghW7Zsue6mNODvXLhwQfbv3++4fujQIfOZ0n0x4eHh4m18rb2AWwODThS//vrrpoKbruNetmyZ3HPPPU6/KJCWplJp2rSp47p9E2PXrl1N/ixv42vtBdwWGMaNGydjx46VMmXKyOzZszMdWgKyQ1el+dJCBF9rL+C2wKBzCbq3oHLlymYISS+ZobQnAARIYOjSpcvfLlcFAARQYGDsFAACA5m/AAAWBAYAgAWBAQBgQWAAAFgQGAAAFgQGAIAFgQEAYEFgAABYEBgAABYEBgCABYEBAGBBYAAA5LyCG+BKY5J+z9bjBtcr6TVt8WSbAVejxwAAsCAwAAAsCAwAAAsCAwDAgsAAALAgMAAALAgMAAALAgMAwILAAACwIDAAACwIDAAACwIDAMCCwAAAsCAwAAAsCAwAAAsCAwDAgsAAALAgMAAALAgMAAALAgMAwILAAACwIDAAALwvMEyePFkqVqwo+fPnl7vvvls2bNjg6SYBQMDyeGCYO3euxMTESHx8vGzevFnq1KkjLVq0kOTkZE83DQACkscDw/jx4+XZZ5+Vbt26SY0aNeTdd9+VggULyrRp0zzdNAAISHk9+eJ//PGHbNq0SYYMGeI4ljt3bnnwwQdl3bp1Ge5/5coVc7E7e/as+ffMmTNy7do1p1//8vlz4i5nzuS96a95o9fNCXe3OZDeqzvaHEg89f9z2QPnCmecO/ff9tlsNhe06L9P5DFHjx7Vd2Fbu3at5fiAAQNsDRo0yHD/+Ph4c38uXLhw4SIZLgcOHHDJudmnvr5oz0LnI+y0l3Dq1CkpUaKE5MqVy62vrRG5fPnycuTIESlSpIj4M96r/wqk9xtI7/Xs2bMSHh4uxYsXd8nzeTQwlCxZUvLkySO//fab5bheL1OmTIb7BwcHm0taRYsWlZtJP2D+/iGz4736r0B6v4H0XnPnzu37k89BQUFy5513yrJlyyy9AL3esGFDTzYNAAKWx4eSdGioa9euctddd0mDBg1kwoQJcvHiRbNKCQAQgIGhU6dOkpKSInFxcXLixAmpW7eufP3111K6dGnxJjqEpXst0g9l+SPeq/8KpPfLe82+XDoDnYPHAwD8jMc3uAEAvAuBAQBgQWAAAFgQGAAAFgSGLAiUtOCjR4+W+vXrS0hIiISFhckjjzwie/bskUAwZswYs3u+X79+4o+OHj0qTz31lMkSUKBAAaldu7Zs3LhR/M3Vq1clNjZWKlWqZN5nRESEjBw50nU5hDxs1apV0rZtWylbtqz5vH766aeW2/V96grPW265xbx/zTu3b98+p1+HwPA3Aikt+MqVK6VXr16yfv16Wbp0qfz555/SvHlzs6/En/3444/y3nvvye233y7+6PTp09K4cWPJly+ffPXVV7Jz50558803pVixYuJvxo4dKwkJCfL222/Lrl27zPVx48bJpEmTxB9cvHjRnIP0y2pm9L1OnDjRZKn+4YcfpFChQuZ8dfnyZedeyCUZl/yYJvPr1auX4/rVq1dtZcuWtY0ePdrm75KTk01irpUrV9r81fnz521VqlSxLV261HbffffZ+vbta/M3gwYNsjVp0sQWCFq3bm3r3r275ViHDh1sUVFRNn8jIraFCxc6rl+7ds1WpkwZ27/+9S/HsTNnztiCg4Nts2fPduq56TFkIS24dseykhbc39jTmrsqMZc30h5S69atLf/H/ubzzz83mQUee+wxM0RYr149mTp1qvijRo0amZQ6e/fuNde3bt0qq1evlpYtW4q/O3TokNkknPazHBoaaoa/nT1feXznszf7/fffzZhl+l3Yen337t3izzRnlY636xBErVq1xB/NmTPHDA/qUJI/O3jwoBle0SHRV155xbzf6Ohok6tM09H4k8GDB5usqpGRkSZBp/79jho1SqKiosTfnThxwvyb2fnKfltWERhw3W/S27dvN9+2/JGmYu7bt6+ZS9FFBf4e5LXH8Prrr5vr2mPQ/1sdh/a3wDBv3jyZNWuWJCYmSs2aNWXLli3mC45O1vrbe3UnhpJcmBbcX/Tu3VsWL14sy5cvl3Llyok/0iFCXUBwxx13SN68ec1FJ9914k5/1m+a/kJXqGjZ3LSqV68uv/zyi/ibAQMGmF5D586dzcqrp59+Wvr3729W3Pm7Mv87J7nifEVguIFASwuu81kaFBYuXCjfffedWfLnr5o1aybbtm0z3yjtF/1WrUMO+rN+IfAXOhyYftmxjsFXqFBB/E1qamqGmgT6f5md0r++Rv9eNQCkPV/psJquTnL6fOXSaXI/NGfOHDOrP2PGDNvOnTttzz33nK1o0aK2EydO2PzNiy++aAsNDbWtWLHCdvz4ccclNTXVFgj8dVXShg0bbHnz5rWNGjXKtm/fPtusWbNsBQsWtH300UeebprLde3a1XbrrbfaFi9ebDt06JDtk08+sZUsWdI2cOBAm7+soktKSjIXPX2PHz/e/Hz48GFz+5gxY8z56bPPPrP99NNPtnbt2tkqVapku3TpklOvQ2DIgkmTJtnCw8NtQUFBZvnq+vXrbf7oenVkp0+fbgsE/hoY1KJFi2y1atUyX3IiIyNtU6ZMsfmjc+fOmf9D/XvNnz+/7bbbbrMNHTrUduXKFZs/WL58eaZ/oxoQ7UtWY2NjbaVLlzb/182aNbPt2bPH6dch7TYAwII5BgCABYEBAGBBYAAAWBAYAAAWBAYAgAWBAQBgQWAAAFgQGAAAFgQG+L3MSiBmxYwZM6Ro0aJuaRPgzQgM8FrPPPOMOanrRctSapKwgQMHOl+mMJs6derkKPiSHZqhVWtJa20Arb+rBY+0aMp//vMfl7YTcDXqMcCrPfTQQzJ9+nRTf1pTZWtOfQ0UWsvXnfT19GSul+waPny4qSWt9Yc1c6tmuty4caOpwezOqoOaFRjICXoM8GrBwcEmlXD58uXlkUceMWULtbiOXcWKFWXChAmWx9StW1deffVVy7Hjx4+b8o56or/ttttk/vz5jtt+/vlnE2zmzp0r9913nynco8VeMhtKWrRokdSvX9/cR+t1tG/f/oYlNXv27GlKampvR4u49+jRQ15++WXHfTQdtBZwr1y5snmv4eHhpuKYnaYGf+CBB0y7S5QoIc8995xcuHDB0qvS34s+RovRVKtWzVGI6PHHHzft155Ku3btzPsEsoLAAJ+hVcfWrl2brW/EsbGx0rFjR1MDWGsuaCGXXbt2We6jBV60qpseb9GiRYbn+OKLL0wgaNWqlSQlJZm89w0aNLjua2pA07oWKSkp173PkCFDzHCTtm/nzp2m8pi9NOPFixdNO4oVK2bKcX788cfy7bffmpoZaWk7tN6CBkwtsKS9HX1cSEiIfP/997JmzRopXLiw6X1pjwL4W65PDAu4hqYSzpMnj61QoUImhbB+XHPnzm2bP3++4z4VKlSwvfXWW5bH1alTxxYfH++4ro974YUXLPe5++67Tf0JpXn79T4TJkyw3EfTjWt9CruGDRvaoqKistz+HTt22KpXr27aXLt2bdvzzz9v+/LLLy0povV9TZ06NdPHa2rsYsWK2S5cuOA49sUXX5jns9cD0d+RplhOm1b6ww8/tFWrVs2kYLbT2wsUKGBbsmRJltuPwEWPAV6tadOmpqKaVqHS+YVu3bqZb/7OSl/BSq+n7zHoPMCNaDu08ltWaTlN7eWsX79eunfvbkqJtm3bVv75z3+a2/X1r1y5ct3n1Nt1+KlQoUKWamw6/JS2IpuWsEzbi9Je0f79+02PQXsKetHhJJ20P3DgQJbbj8DF5DO8mp4UdfxdTZs2zZwo33//fTNWr7SMY/qSIjqUkt3XupHsTERr+3ROQi9alP6jjz4ydYiHDh2ao4ntG7Vb5yC0JK3Ok6RXqlQpl7wm/Bs9BvgMPcm+8sorMmzYMLl06ZLjRKcTy3a68ufQoUMZHqvf2tNfr169ulOvf/vtt1vq6WaH9iLs8wdVqlQxweF6z6nt02//el87nS/Q34N9kjkzd9xxh+zbt0/CwsJMUE17CQ0NzVH7ERgIDPApusJHi7tPnjzZXNcVOx9++KGZZNUVPDrcpLenpxO32uPQfQnx8fGyYcOGDJO4f0cfN3v2bPOvDvPo691o2eyjjz4qb731lhkGO3z4sKxYsUJ69eolVatWNXsbdGXToEGDzN6MDz74wAzzaMDSHpHSSXK9j74nHZJavny59OnTx/Q47BPUmdHH6YopXYmkvxcNlPra0dHR8uuvvzr1nhGYCAzwKXnz5jUndF3iqd+kdVWPLjFt06aNtG7d2izdjIiIyHRPwZw5c8y3fj0J6wne/u09q+6//34TYHQZqi6J1aCkAeZ6dGWQLm/VeQUNBnqC14DwzTffmPehdDXSSy+9JHFxcaaHoJvqdC5CFSxYUJYsWSKnTp0yQ1EaaHQ+QvdF3Ig+btWqVWbpa4cOHczz6tCbzjEUKVLEqfeMwETNZwCABT0GAIAFgQEAYEFgAABYEBgAABYEBgCABYEBAGBBYAAAWBAYAAAWBAYAgAWBAQBgQWAAAEha/w+JpFkGXHXCOwAAAABJRU5ErkJggg==", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA90AAAJOCAYAAACqS2TfAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQCRJREFUeJzt3QmcVXXdP/DvALKogIIgEotkGiGLW/q47xqpqfm4RYlLZuWOG2SAuIGmZiph+pTk3yzNPXvEjNwwTVFcUBNRcokQXFkUUOb+X7/Tc6eZYR2cw52Z+36/XseZe+72vfeei/M5v62iUCgUAgAAAKh3zer/IQEAAIBE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5EToBmCZzjvvvKioqFgjz7XbbrtlW9FDDz2UPfdtt922Rp7/6KOPjo033jgasvnz58d3v/vd6NKlS/benHbaadHQ/OMf/8hqGz9+fDRkxeMr/QSAvAndAGUghaAUMopb69ato2vXrrHvvvvGVVddFfPmzauX55k5c2YW1p999tloaBpybavi4osvzj7HH/zgB/H//t//i+985zvLvW06gVD9866+fe1rX4ty8fOf/7zBnQCorKyMG2+8Mbbbbrvo0KFDtG3bNjbbbLM46qij4oknnih1eQDkoEUeDwpAw3T++edHr1694tNPP41Zs2ZlLX2pxfSKK66Ie+65J/r371912x//+McxdOjQOgfbUaNGZaFviy22WOX7/elPf4q8rai266+/PgtDDdlf/vKX+K//+q8YOXLkKt0+vcYzzjhjqf3pZEs5he4NNtgg68lQ3S677BKffPJJtGzZco3XdMopp8TYsWPjwAMPjEGDBkWLFi3ilVdeifvuuy+++MUvZp8xAE2L0A1QRgYOHBjbbLNN1eVhw4ZlYW7//fePb3zjG/Hyyy9HmzZtsutSGEhbnj7++ONYe+21SxJ+qltrrbWioZs9e3b06dNnlW//hS98Ib797W/nWlNj1axZs6y3x5r2zjvvZCcCjj/++LjuuutqXHfllVfGnDlz1lgtn332WXaiqdTfPYByoHs5QJnbY489Yvjw4fHGG2/ETTfdtMIx3Q888EDstNNOsd5668W6664bX/7yl+NHP/pRdl1qNf/qV7+a/X7MMcdUdWcudu9NY7b79u0bTz/9dNbSmMJ28b61x3QXLVmyJLtNGse8zjrrZCcG3nrrrRq3SS3XtVsyaz/mympb1pjuBQsWZC3F3bt3j1atWmWv9bLLLotCoVDjdulxTjrppLjrrruy15duu/nmm8eECRNWOUwfd9xxseGGG2ZBcMCAAfHrX/96qfHHM2bMiD/+8Y9Vtafx059Het5OnTpl71H11zR9+vTsvT788MOr9lX/7HbYYYfsxEzqMXHttdeu9Hmef/757P1Nrbjp9aXP8thjj4333nuvxu2Kx1t6/nT7dIy1b98++7zSyZnqbrjhhuy47dy5c/Z+p5MR48aNq3Gb9Hm++OKL8fDDD1e9Z9WPh2WN6f79738fW2+9dfb6Ugt5Omnxz3/+s8ZtUm3p2E/7DzrooOz39D6eeeaZ2fG6IukzTO/1jjvuuNR1qZ70eqr78MMP4/TTT89eS3qd3bp1y7qhv/vuu6t8/FQfa5+O3xTuN9lkk+zxXnrppez6v//97/Hf//3fWXf39BjpxFzq+VJd6h2Teopsuumm2W06duyY/VuQ/k0AYMW0dAOQjQ9O4TZ1806tcMuSAkxqEU9d0FM39fRHewpIjz32WHb9V77ylWz/iBEj4nvf+17svPPO2f4U0opS0Eqt7UcccUQWaFJQWJGLLrooCwvnnHNOFi5SYNhrr72ycdnFFvlVsSq1VZeCUQr4Dz74YBZoUlft+++/P84666wsbP30pz+tcftJkybFHXfcET/84Q+zMbppnPwhhxwSb775ZhZOlid1cU5BML2PKbinIJuCXwp2KXCdeuqpWe1pDHcKXyl0FbuMp6C3IikkVQ9nRSlQp/cuBbwUVA899NC4+uqrs27PqeUzPXd6DalFtroPPvggvv71r8dhhx0WRx55ZNx6663Z+PLUUppC9PKkUPb6669n4TkF7nQcpVbe9DONYa59Yic9fnofRo8eHc8880z8z//8T1brJZdcUnWbVHc6sZE+o9Qb4w9/+EP23qf6TzzxxOw26Vg5+eSTs1B87rnnZvtWdLylEzCpxnRyJj13apX+2c9+lh3fU6ZMyU4CFKVwneZDSOOyU5D985//HJdffnkWZtN7sjw9e/bMfqbPOL3v6cTTiibOS8dp6n2S3t+tttoq+zxTGH777bezkwKrcvzUPlmxcOHC7DuQvr8pZKfPIZ0ESD0j0nCSdHykzzadULj99tvj4IMPrjopkt6XNJnftttuG3Pnzo3Jkydnn9Hee++93NcBwL//sACgibvhhhtSU2bhqaeeWu5t2rdvX9hyyy2rLo8cOTK7T9FPf/rT7PKcOXOW+xjp8dNt0vPVtuuuu2bXXXvttcu8Lm1FDz74YHbbL3zhC4W5c+dW7b/11luz/T/72c+q9vXs2bMwePDglT7mimpL90+PU3TXXXdlt73wwgtr3O6///u/CxUVFYXp06dX7Uu3a9myZY19zz33XLb/6quvLqzIlVdemd3upptuqtq3ePHiwvbbb19Yd911a7z2VN9+++23wserftv0uMvaRo8eXeO2Rx55ZGHttdcuTJs2rfCTn/wku016/cv67C6//PKqfYsWLSpsscUWhc6dO2c1JzNmzFjqPf7444+Xqu+3v/1tdrtHHnlkqePt2GOPrXHbgw8+uNCxY8ca+5b1mPvuu2/hi1/8Yo19m2++eY1joPbxlX4mqf70Ovr27Vv45JNPqm537733ZrcbMWJEjWMl7Tv//PNrPGb67my99daFlTnqqKOy+6+//vrZa7vssssKL7/88lK3S8+ZbnfHHXcsdV1lZWWdjp/i59KuXbvC7NmzazzWnnvuWejXr19h4cKFNR5/hx12KGy66aZV+wYMGLDKxx8ANeleDkAmtQiuaBbzYkvf3XffvdqTjqXWtdSauKpSV9rU6lqUusButNFG8b//+7+Rp/T4zZs3z1p/q0utzClnp0mvqkut76mVsyj1BmjXrl3Wwruy50mtv6nluPr48vS8qaUzdY1eXakVNrUy196qP1dyzTXXZN2403ubhhmkXg9pkq/aUovyCSecUHU5tXCny6kHQup2vjzVeySkVtbUWlucLCy1ktb2/e9/v8bl1NqbekikltVlPeZHH32UPeauu+6avd/pcl2lFtv0OlJrefWx3vvtt1/07t0769a/KnWu7PMutjan9zy1St95551Zt/TUm2HPPfes0ZU9tTKnruLFlubqir0D6nr8pN4X1XtIvP/++9mcDql3Qfrup/cxben9Ti35r776alVN6fufWsXTPgDqRugGIJP+SK8ecGtLY3xTN9TUvTR1001dxFM31LoE8NSFtS4TN6Xxo7XDxpe+9KXPPZ55ZdL49jTLd+33I4Wj4vXV9ejRY6nHWH/99bMu2St7nvQa08Req/I8dZG6H6eTAbW3YhfnotTFOHWHT2OvU/hOvy9Lej9S1+Pq0lJXyYo+jxTsUjfndMyksJxCXwqcybICcu33Mr2PSfX3MnX5Tq8l1ZPCYHrM4vwAqxO6i+9zGrdfWwrdtT+HFMxrd+9flc87SZ916gKfTlSkgJtOYqUhFyn8pu9U0WuvvZaNo6/P46f4vhelbunpJFI62ZJeT/WtOEt+OhmRpOEZqct6+sz79euXDbVIxwwAK2dMNwDZGNEUVlKgXZ4UmB555JFsnHNq+UsThd1yyy3ZhFZpLHhqGV6ZuozDXlW1xwRXH3e7KjXVh+U9T+1J1xqqNF49SaExHQvVxy9/XqkV9a9//WsW0tLY+NSjIp2oSeuFL+uEzcreyxRGU6twCsNpqbs00V06kZNafdNY+zWx9Ft9HVdpvH8al562NDY7tUynoFz7xEh9qf39K75XqbU9tWwvS/HfhDT5YXrv00mC9H1PY+3T+50m00sn4gBYPi3dAGQTdSXL+8O7KLWopcCTwk6a+ThNdJZa6FIQX1EAXl21u7Km4JVa56rPNJ5aGFMLXG21W/nqUlsKPWld79rd7dMsz8Xr60N6nPQaawfF+n6eFUknT1KAOvvss7MWzsGDB2fLSdWW3o80o3t106ZNy37Wnvm9KIX4iRMnZhN0pZmvU1fpNOlWmsl8daVJ0xYtWpRNKJa6t6fJ3VKr97JO6KzqZ158n9N62bWlfWvicygu5fevf/0r+5mGK0ydOjXX46f4OaQu6cvqFZG26r09Uq+INDzkt7/9bbaKQBpGkSZYA2DFhG6AMpdC8wUXXJB1PR00aNAKuwnXlloukxSCkmL342WF4NVx44031gi+t912WxZKUnfcohRO0izYixcvrtp37733LrW0WF1qS0EutZSnsbfVpZa9FOSqP//nkZ5n1qxZWY+BohR402ziqUU4jVPOU3ovirNRX3zxxVn4TuOs0++1pbp+8YtfVF1O73e6nIJ6WmZrRS3CtVv808ziq2tZj5l6aaSx0rWlz3xVPu8UeNMM6anVtngsJ2nsfpo9PI3trg/psy4u01Vdei/TyYl0UqvYspzGXz/33HPZuO/aiq/98x4/6TWnFvb0ORbDfnXV1w2vvcRbevxUa/X3C4Bl0728jKRuoT/5yU+ycWTpf67pf+RpSZCitNxN+oMjXZ/+uE5LpBT/oAaahhQiUitY+sM8LYmUAneaXCu1iKWWw+qTSNWWxnSmf0dSAEm3T2M907JSaRmrtF5vMQCnrsnp35LUQpZCT5rQq/ZY0lWVWtbSY6fWtVRvCmvpD/3qy5ql0JjCeOqunLoypy6wab3x6hOb1bW2Aw44IHbfffdsqak0XjlNaJW61KautaeddtpSj7260tJNKfCkJZ7Sv72pxTi9ljRmOb3WFY2xX5k0AVb1dderh6Xiv/1prHUKU2nJqxRm03uY3s8LL7wwm0wtve7qY7rTsl3p/UjjelPQS0u3peW/UkvpsqTJ5FK35EsvvTRbwiyN6U/vY1qvenXts88+WXfy9Bmllu40F8H111+fBcjawTGdDEjLi6XXk46bdJs0HKK2VH96bek4S0E1TUxWXDIsfSZpubb6kLrupxMcqYbUYyRNgpa+R6nlOAXsdGylsfhJ6o6fjoW0tFhaMiy9lvT/5vQ9Tcdw+mzq4/gZO3Zs9h1L47TT9yq1fqfX/vjjj2f1prqStBZ6CuipjvS9TJPPpedKS5UBsBK1ZjOnCfvf//3fwrnnnpstP5I++jvvvLPG9TfeeGNh1KhRheuvvz67fsqUKSWrFchnybDilpa46tKlS2HvvffOlt+qvjTV8pYMmzhxYuHAAw8sdO3aNbt/+pmWm0pLTVV39913F/r06VNo0aJFjeWj0tJNaQmnZVnekmFpaalhw4Zlyzm1adMmW7LojTfeWOr+aSmrtLxYq1atCjvuuGNh8uTJSz3mimqrvWRYMm/evMLpp5+evc611lorWz4pLalVXK6pKD3OiSeeuFRNy1vKrLZ33nmncMwxxxQ22GCD7H1Nyzcta1mz+loyrPg603tRexmwJB0L6TZpiajiUmDFzy69r2k5qtatW2e3ueaaa2rcd1lLhr399tvZ0ljrrbdetizdoYceWpg5c2Z2u3SM1T7eai9JVzx202MX3XPPPYX+/ftndWy88caFSy65pPCrX/1qqdvNmjUre8/atm2bXVc8HmovGVZ0yy23ZEt/peOoQ4cOhUGDBmX1V5c+03XWWWel35dlSe9t+r6l5c26deuWHVeptvSepv/31j623nvvvcJJJ52UHdvp2Ej3Sc//7rvv1un4KX4u6fhdltdeey1byiz9m5BqSs+3//77F2677baq26Tl87bddtvsc0zfxd69excuuuiiqmMEgOWrSP9ZWTCn6UndI2u3dBelVozU8qOlG4AktXCmmbZXNsYYAFiaMd0AAACQE6EbAAAAciJ0AwAAQE7MXg4ArNBDDz1U6hIAoNHS0g0AAAA50dJdRtJaptOnT6+6nNZJTWuspvU2e/Toka3/+eabb8bMmTOz61955ZXsZ1pHNG0AAADUTZNfMqyysjILkW3bts2WySpnjz76aOy///5L7f/Wt74V48aNi9/85jfxwx/+cKnrhw4dGsOGDVtDVQIAADR8KUrPmzcvunbtGs2aNSvf0P32229H9+7dS10GAAAATdBbb70V3bp1K9/u5amFO0lvRLt27UpdDgAAAE3A3LlzswbeYuYs29Bd7FKeArfQDQAAQH1a2TBms5cDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5KRFXg9M3Y2Z8m6Uo6FbblDqEgAAAHKhpRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5EToBgAAgJwI3QAAAJAToRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICmGLofeeSROOCAA6Jr165RUVERd911V43rC4VCjBgxIjbaaKNo06ZN7LXXXvHqq6+WrF4AAABoNKF7wYIFMWDAgBg7duwyr7/00kvjqquuimuvvTb+9re/xTrrrBP77rtvLFy4cI3XCgAAAHXVIkpo4MCB2bYsqZX7yiuvjB//+Mdx4IEHZvtuvPHG2HDDDbMW8SOOOGINVwsAAACNKHSvyIwZM2LWrFlZl/Ki9u3bx3bbbRePP/74ckP3okWLsq1o7ty52c/Kyspsa9AKhShHDf5zAQAAWM0c02BDdwrcSWrZri5dLl63LKNHj45Ro0YttX/OnDkNvlv6up/8+wRBuZk9e0mpSwAAAKiTefPmNe7QvbqGDRsWQ4YMqdHS3b179+jUqVO0a9cuGrL5M5tHOercuWOpSwAAAKiT1q1bN+7Q3aVLl+znO++8k81eXpQub7HFFsu9X6tWrbKttmbNmmVbg1ZREeWowX8uAAAAq5ljGmza6dWrVxa8J06cWKPVOs1ivv3225e0NgAAAGjwLd3z58+P6dOn15g87dlnn40OHTpEjx494rTTTosLL7wwNt100yyEDx8+PFvT+6CDDipl2QAAANDwQ/fkyZNj9913r7pcHIs9ePDgGD9+fJx99tnZWt7f+9734sMPP4yddtopJkyYsMp95wEAAKCUKgppQewmLHVJT0uNffTRRw1+IrUxU96NcjR0yw1KXQIAAEAuWbPBjukGAACAxk7oBgAAgJwI3QAAAJAToRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5EToBgAAgJwI3QAAAJAToRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5EToBgAAgJwI3QAAAJAToRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5EToBgAAgJwI3QAAAJAToRsAAADKMXQvWbIkhg8fHr169Yo2bdrEJptsEhdccEEUCoVSlwYAAAAr1SIasEsuuSTGjRsXv/71r2PzzTePyZMnxzHHHBPt27ePU045pdTlAQAAQOMN3X/961/jwAMPjP322y+7vPHGG8dvf/vbePLJJ0tdGgAAADTu0L3DDjvEddddF9OmTYvNNtssnnvuuZg0aVJcccUVy73PokWLsq1o7ty52c/Kyspsa9DKtNt8g/9cAAAAVjPHNOjQPXTo0Cw09+7dO5o3b56N8b7oooti0KBBy73P6NGjY9SoUUvtnzNnTixcuDAasnU/+fcJgnIze/aSUpcAAABQJ/PmzWv8ofvWW2+N3/zmN3HzzTdnY7qfffbZOO2006Jr164xePDgZd5n2LBhMWTIkKrLKbR37949OnXqFO3atYuGbP7M5lGOOnfuWOoSAAAA6qR169aNP3SfddZZWWv3EUcckV3u169fvPHGG1lr9vJCd6tWrbKttmbNmmVbg1ZREeWowX8uAAAAq5ljGnTa+fjjj5d6IambuTHAAAAANAYNuqX7gAMOyMZw9+jRI+tePmXKlGwStWOPPbbUpQEAAEDjDt1XX311DB8+PH74wx/G7Nmzs7HcJ5xwQowYMaLUpQEAAEDjDt1t27aNK6+8MtsAAACgsWnQY7oBAACgMRO6AQAAICdCN1DWNt5446ioqFhqO/HEE0tdGgAATUCDHtMNkLennnoqlixZUnV56tSpsffee8ehhx5a0roAAGgahG6grHXq1KnG5TFjxsQmm2wSu+66a8lqAgCg6dC9HOD/LF68OG666aY49thjsy7mAADweQndAP/nrrvuig8//DCOPvroUpcCAEATIXQD/J9f/vKXMXDgwOjatWupSwEAoIkwphsgIt54443485//HHfccUepSwEAoAnR0g0QETfccEN07tw59ttvv1KXAgBAEyJ0A2WvsrIyC92DBw+OFi10AAIAoP4I3UDZS93K33zzzWzWcgAAqE+adICyt88++0ShUCh1GQAANEFaugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBDCd0TJkyISZMmVV0eO3ZsbLHFFvGtb30rPvjgg/quDwAAAMondJ911lkxd+7c7PcXXnghzjjjjPj6178eM2bMiCFDhuRRIwAAADRKLep6hxSu+/Tpk/1+++23x/777x8XX3xxPPPMM1n4BgAAAFazpbtly5bx8ccfZ7//+c9/jn322Sf7vUOHDlUt4AAAAMBqtHTvtNNOWTfyHXfcMZ588sm45ZZbsv3Tpk2Lbt265VEjAAAAlEdL9zXXXBMtWrSI2267LcaNGxdf+MIXsv333XdffO1rX8ujRgAAACiPlu4ePXrEvffeu9T+n/70p/VVEwAAAJTvOt2vvfZa/PjHP44jjzwyZs+eXdXS/eKLL9Z3fQAAAFA+ofvhhx+Ofv36xd/+9re44447Yv78+dn+5557LkaOHJlHjQAAAFAeoXvo0KFx4YUXxgMPPJDNZF60xx57xBNPPFHf9QEAAED5hO4XXnghDj744KX2d+7cOd599936qgsAAADKL3Svt9568a9//Wup/VOmTKmayRwAAABYjdB9xBFHxDnnnBOzZs2KioqKqKysjMceeyzOPPPMOOqoo/KpEgAAAMohdF988cXRu3fv6N69ezaJWp8+fWKXXXaJHXbYIZvRHAAAAFjNdbrT5GnXX399DB8+PKZOnZoF7y233DI23XTTuj4UAAAANGl1Dt1FPXr0yDYAAADgc4TuIUOGxKq64oorVvm2AAAAEOUeutPM5KsiTawGAAAA1CF0P/jgg6tyM4DPbcyUd6McDd1yg1KXAABAQ5i9vLq33nor2wAAAIB6CN2fffZZNnN5+/btY+ONN8629HtaLuzTTz+t68MBAABAk1Xn2ctPPvnkuOOOO+LSSy+N7bffPtv3+OOPx3nnnRfvvfdejBs3Lo86AQAAoOmH7ptvvjl+97vfxcCBA6v29e/fP7p37x5HHnmk0A0AAACr2728VatWWZfy2nr16hUtW7as68MBAABAk1Xn0H3SSSfFBRdcEIsWLaral36/6KKLsusAAACA1exentbsnjhxYnTr1i0GDBiQ7Xvuuedi8eLFseeee8Y3v/nNqtumsd8AAABQruocutdbb7045JBDauxL47kBAACAzxm6b7jhhrreBQAAAMpSncd0AwAAADm1dKe1uEeMGBEPPvhgzJ49OyorK2tc//7779f1IQEAAKBJqnPo/s53vhPTp0+P4447LjbccMOoqKjIpzIAAAAot9D96KOPxqRJk6pmLgcAAADqaUx3796945NPPqnr3QAAAKDs1Dl0//znP49zzz03Hn744Wx899y5c2tsAAAAwOdYpzuF6z322KPG/kKhkI3vXrJkSV0fEgAAAJqkOofuQYMGxVprrRU333yzidQAAACgPkP31KlTY8qUKfHlL3+5rncFAACAslLnMd3bbLNNvPXWW/lUAwAAAOXc0n3yySfHqaeeGmeddVb069cv62peXf/+/euzPgAAACif0H344YdnP4899tiqfWlct4nUAAAA4HOG7hkzZtT1LgAAAFCW6hy6e/bsmU8lAAAAUO6hu+ill16KN998MxYvXlxj/ze+8Y36qAsAAADKL3S//vrrcfDBB8cLL7xQNZY7Ka7XbUw3AAAArOaSYWnm8l69esXs2bNj7bXXjhdffDEeeeSRbCmxhx56qK4PBwAAAE1WnVu6H3/88fjLX/4SG2ywQTRr1izbdtpppxg9enSccsopMWXKlHwqBQAAgKbe0p26j7dt2zb7PQXvmTNnVk2w9sorr9R/hQAAAFAuLd19+/aN5557Lutivt1228Wll14aLVu2jOuuuy6++MUv5lMlAAAAlEPo/vGPfxwLFizIfj///PNj//33j5133jk6duwYt9xySx41AgAAQHmE7n333bfq9y996Uvx97//Pd5///1Yf/31q2YwBwAAAFZjTPecOXOW2tehQ4cscKdlxAAAAIDVDN39+vWLP/7xj0vtv+yyy2Lbbbet68MBAABAk1Xn0D1kyJA45JBD4gc/+EF88skn8c9//jP23HPPbEK1m2++OZ8qAQAAoBxC99lnn52t1f3oo49G//79s61Vq1bx/PPPx8EHH5xPlQAAAFAOobs4gVpaOuwf//hHzJ07Nw4//PDo0qVL/VcHAAAA5RS6H3vssax1+9VXX81at8eNGxcnn3xyFrw/+OCDfKoEAACAcgjde+yxRxawn3jiifjKV74S3/3ud2PKlCnx5ptvZpOsAQAAAKsZuv/0pz/FmDFjYq211qrat8kmm2Qt4CeccELUtzRR27e//e3o2LFjtGnTJgv2kydPrvfnAQAAgPrWoq532HXXXZe5v1mzZjF8+PCoT6m7+o477hi777573HfffdGpU6esW/v6669fr88DAAAAJW3p/vrXvx4fffRR1eXU2v3hhx9WXX7vvfeiT58+9VrcJZdcEt27d48bbrghWwO8V69esc8++2Qt6wAAANBkQvf9998fixYtqrp88cUXx/vvv191+bPPPotXXnmlXou75557YptttolDDz00OnfuHFtuuWVcf/319focAAAAUPLu5YVCYYWX8/D6669ns6MPGTIkfvSjH8VTTz0Vp5xySrRs2TIGDx68zPukEwPVTw6kJc2SysrKbGvQ1sB72hA1+M+FNcv3AACAJvT3W53HdK/pF5FaulOrepJauqdOnRrXXnvtckP36NGjY9SoUUvtnzNnTixcuDAasnU/+fcJgnIze/aSUpdAA+J7AABAYzBv3rz6Dd0VFRXZVntfnjbaaKOlxomnZcpuv/325d5n2LBhWct49ZbuNC48TcLWrl27aMjmz2we5ahz546lLoEGxPcAAIDGoHXr1vXfvfzoo4+OVq1aZZdTq/H3v//9WGeddbLL1bt015c0c3ntceLTpk2Lnj17Lvc+qb5ijbVnV09bg5bzSYyGqsF/LqxZvgcAADShv99WOXTX7s6d1s6u7aijjor6dPrpp8cOO+yQdS8/7LDD4sknn4zrrrsu2wAAAKChW+XQnZbtWtO++tWvxp133pl1GT///POzJcOuvPLKGDRo0BqvBQAAAOqqQU+kluy///7ZBgAAAI2NQYQAAACQE6EbAAAAciJ0AwAAQClD91ZbbRUffPBB9nua0Ozjjz/Oqx4AAABK5LzzzouKiooaW+/evUtdVtMP3S+//HIsWLAg+33UqFExf/78vOsCAACgBDbffPP417/+VbVNmjSp1CU1/dnLt9hiizjmmGNip512ikKhEJdddlmsu+66y7ztiBEj6rtGAAAA1pAWLVpEly5dSl1GeYXu8ePHx8iRI+Pee+/Nuhfcd9992QdRW7pO6AYAAGi8Xn311ejatWu0bt06tt9++xg9enT06NGj1GU17dD95S9/OX73u99lvzdr1iwmTpwYnTt3zrs2AAAA1qDtttsua3RNGTB1LU/Di3feeeeYOnVqtG3bttTlNd3QXV1lZWU+lQAAAFBSAwcOrPq9f//+WQjv2bNn3HrrrXHccceVtLayCd3Ja6+9FldeeWU2wVrSp0+fOPXUU2OTTTap7/oAAAAokfXWWy8222yzmD59eqlLKZ91uu+///4sZD/55JPZmY+0/e1vf8tmuHvggQfyqRIAAIA1Lq1clRpdN9poo1KXUj4t3UOHDo3TTz89xowZs9T+c845J/bee+/6rA8AAIA15Mwzz4wDDjgg61I+c+bMbELt5s2bx5FHHlnq0sqnpTt1KV9WX/5jjz02XnrppfqqCwAAgDXs7bffzgJ2mkjtsMMOi44dO8YTTzwRnTp1KnVp5dPSnd7sZ599NjbddNMa+9M+M5oDAAA0XsVVqyhh6D7++OPje9/7Xrz++uuxww47ZPsee+yxuOSSS2LIkCH1WBoAAACUWegePnx4tj7b5ZdfHsOGDcv2pYXTzzvvvDjllFPyqBEAAADKI3RXVFRkE6mlbd68edk+i6QDAABAPa3TXSRsAwAAQD3OXg4AAACsGqEbAAAAciJ0AwAAQEMI3Z9++mnsueee8eqrr+ZVDwAAAJRn6F5rrbXi+eefz68aAAAAKOfu5d/+9rfjl7/8ZT7VAAAAQDkvGfbZZ5/Fr371q/jzn/8cW2+9dayzzjo1rr/iiivqsz4AAAAon9A9derU2GqrrbLfp02bVuO6ioqK+qsMAAAAyi10P/jgg/lUAgAAAE3Mai8ZNn369Lj//vvjk08+yS4XCoX6rAsAAADKL3S/99572bJhm222WXz961+Pf/3rX9n+4447Ls4444w8agQAAIDyCN2nn356tnTYm2++GWuvvXbV/sMPPzwmTJhQ3/UBAABA+Yzp/tOf/pR1K+/WrVuN/Ztuumm88cYb9VkbAAAAlFdL94IFC2q0cBe9//770apVq/qqCwAAAMovdO+8885x44031lgmrLKyMi699NLYfffd67s+AAAAKJ/u5Slcp4nUJk+eHIsXL46zzz47Xnzxxayl+7HHHsunSgAAACiHlu6+ffvGtGnTYqeddooDDzww627+zW9+M6ZMmRKbbLJJPlUCAABAObR0J+3bt49zzz23/qsBAACAcg/dH3zwQfzyl7+Ml19+Obvcp0+fOOaYY6JDhw71XR8AAACUT+h+5JFH4oADDshau7fZZpts31VXXRXnn39+/OEPf4hddtkljzoBAADKzpgp70Y5GrrlBlG2ofvEE0+Mww8/PMaNGxfNmzfP9i1ZsiR++MMfZte98MILedQJAAAATX8itenTp8cZZ5xRFbiT9PuQIUOy6wAAAIDVDN1bbbVV1Vju6tK+AQMG1PXhAAAAoMlape7lzz//fNXvp5xySpx66qlZq/Z//dd/ZfueeOKJGDt2bIwZMya/SgEAAKAphu4tttgiKioqolAoVO07++yzl7rdt771rWy8NwAAALCKoXvGjBn5VwIAAADlGLp79uyZfyUAAADQxNR5ybBk5syZMWnSpJg9e3ZUVlbWuC6N+QYAAABWI3SPHz8+TjjhhGjZsmV07NgxG+tdlH4XugEAAGA1Q/fw4cNjxIgRMWzYsGjWrM4rjgEAAEDZqHNq/vjjj+OII44QuAEAAGAl6pycjzvuuPj9739f17sBAABA2alz9/LRo0fH/vvvHxMmTIh+/frFWmutVeP6K664oj7rAwAAgPIK3ffff398+ctfzi7XnkgNAAAAWM3Qffnll8evfvWrOProo+t6VwAAACgrdR7T3apVq9hxxx3zqQYAAADKOXSfeuqpcfXVV+dTDQAAAJRz9/Inn3wy/vKXv8S9994bm2+++VITqd1xxx31WR8AAACUT+heb7314pvf/GY+1QAAAEA5h+4bbrghn0oAAACg3Md0AwAAADm1dPfq1WuF63G//vrrdX1IAAAAaJLqHLpPO+20Gpc//fTTmDJlSkyYMCHOOuus+qwNAAAAyit0pyXDlmXs2LExefLk+qgJAAAAmoR6G9M9cODAuP322+vr4QAAAKDRq7fQfdttt0WHDh3q6+EAAACg/LqXb7nlljUmUisUCjFr1qyYM2dO/PznP6/v+gAAAKB8QvdBBx1U43KzZs2iU6dOsdtuu0Xv3r3rszYAAAAor9A9cuTIfCoBAACAJqbexnQDAAAAq9nSnbqRVx/LvSzp+s8++2xVHxIAAACatFUO3Xfeeedyr3v88cfjqquuisrKyvqqCwAAAMondB944IFL7XvllVdi6NCh8Yc//CEGDRoU559/fn3XBwAAAOU1pnvmzJlx/PHHR79+/bLu5M8++2z8+te/jp49e9Z/hQAAAFAOofujjz6Kc845J770pS/Fiy++GBMnTsxaufv27ZtfhQAAANDUu5dfeumlcckll0SXLl3it7/97TK7mwMAAACrEbrT2O02bdpkrdypK3naluWOO+5Y1YcEAACAJm2VQ/dRRx210iXDAAAAgNUI3ePHj1/VmwIAAACrO3t5qYwZMyZrbT/ttNNKXQoAAAA0ndD91FNPxS9+8Yvo379/qUsBAACAphO658+fH4MGDYrrr78+1l9//VKXAwAAAPU7pruUTjzxxNhvv/1ir732igsvvHCFt120aFG2Fc2dOzf7WVlZmW0NWqEQ5ajBfy6sWb4HAAD/4W+jRl9jgw/dv/vd7+KZZ57JupevitGjR8eoUaOW2j9nzpxYuHBhNGTrfvLvEwTlZvbsJaUugQbE9wAA4D/8bdRwzZs3r/GH7rfeeitOPfXUeOCBB6J169ardJ9hw4bFkCFDarR0d+/ePTp16hTt2rWLhmz+zOZRjjp37ljqEmhAfA8AAP7D30YN16pm1AYdup9++umYPXt2bLXVVlX7lixZEo888khcc801WTfy5s1rHoStWrXKttqaNWuWbQ1ama6D3uA/F9Ys3wMAgP/wt1Gjr7FBh+4999wzXnjhhRr7jjnmmOjdu3ecc845SwVuAAAAaEgadOhu27Zt9O3bt8a+ddZZJzp27LjUfgAAAGhoGn6bPQAAADRSDbqle1keeuihUpcAAAAAq0RLNwAAAORE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5EToBgAAgJwI3QAAAJAToRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5EToBgAAgJwI3QAAAJAToRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5EToBgAAgJwI3QAAAJAToRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICcCN0AAACQE6EbAAAAyjF0jx49Or761a9G27Zto3PnznHQQQfFK6+8UuqyAAAAoPGH7ocffjhOPPHEeOKJJ+KBBx6ITz/9NPbZZ59YsGBBqUsDAACAlWoRDdiECRNqXB4/fnzW4v3000/HLrvsUrK6AAAAoNGH7to++uij7GeHDh2We5tFixZlW9HcuXOzn5WVldnWoBUKUY4a/OfCmuV7AADwH/42avQ1NprQnV7QaaedFjvuuGP07dt3hePAR40atdT+OXPmxMKFC6MhW/eTf58gKDezZy8pdQk0IL4HAAD/4W+jhmvevHlNK3Snsd1Tp06NSZMmrfB2w4YNiyFDhtRo6e7evXt06tQp2rVrFw3Z/JnNoxx17tyx1CXQgPgeAAD8h7+NGq7WrVs3ndB90kknxb333huPPPJIdOvWbYW3bdWqVbbV1qxZs2xr0Coqohw1+M+FNcv3AADgP/xt1OhrbNChu1AoxMknnxx33nlnPPTQQ9GrV69SlwQAAACrrEVD71J+8803x913352t1T1r1qxsf/v27aNNmzalLg8AAABWqEG32Y8bNy6bsXy33XaLjTbaqGq75ZZbSl0aAAAANP7u5QAAANBYNeiWbgAAAGjMhG4AAADIidANAGTLch5wwAHRtWvXqKioiLvuuqvUJQFAkyB0AwCxYMGCGDBgQIwdO7bUpQBAk9KgJ1IDANaMgQMHZhsAUL+0dAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAOTE7OUAQMyfPz+mT59edXnGjBnx7LPPRocOHaJHjx4lrQ0AGjOhGwCIyZMnx+677151eciQIdnPwYMHx/jx40tYGQA0bkI3ABC77bZbFAqFUpcBAE2OMd0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5EToBgAAgJwI3QAAAJAToRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICctMjrgQGA1TNmyrtRroZuuUGpSwCAeqWlGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAA5EToBgAAgJwI3QAAAJAToRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAICcCN0AAACQE6EbAAAAciJ0AwAAQE6EbgAAAMiJ0A0AAAA5EboBAAAgJ0I3AAAAlHPoHjt2bGy88cbRunXr2G677eLJJ58sdUkAAADQ+EP3LbfcEkOGDImRI0fGM888EwMGDIh99903Zs+eXerSAAAAoHGH7iuuuCKOP/74OOaYY6JPnz5x7bXXxtprrx2/+tWvSl0aAAAANN7QvXjx4nj66adjr732qtrXrFmz7PLjjz9e0toAAABgZVpEA/buu+/GkiVLYsMNN6yxP13++9//vsz7LFq0KNuKPvroo+znhx9+GJWVldGQLZw3N8rRhx826MOQNcz3AMr3e5D4LgDUVK7/T/iwEfz/YO7cf382hUJhhbdr+K+kjkaPHh2jRo1aan/Pnj1LUg8rt/SnBeXH9wD+zXcBgMb2/4N58+ZF+/btG2fo3mCDDaJ58+bxzjvv1NifLnfp0mWZ9xk2bFg28VpRat1+//33o2PHjlFRUZF7zY1ROkPTvXv3eOutt6Jdu3alLocmzLHGmuR4Y01xrLEmOd5YUxxrK5dauFPg7tq16wpv16BDd8uWLWPrrbeOiRMnxkEHHVQVotPlk046aZn3adWqVbZVt956662Rehu79GXyhWJNcKyxJjneWFMca6xJjjfWFMfaiq2ohbtRhO4ktVoPHjw4ttlmm9h2223jyiuvjAULFmSzmQMAAEBD1uBD9+GHHx5z5syJESNGxKxZs2KLLbaICRMmLDW5GgAAADQ0DT50J6kr+fK6k/P5pe74I0eOXKpbPtQ3xxprkuONNcWxxprkeGNNcazVn4rCyuY3BwAAAFZLs9W7GwAAALAyQjcAAADkROgGAACAnAjdZW7s2LGx8cYbR+vWrWO77baLJ598stQl0QSNHj06vvrVr0bbtm2jc+fOcdBBB8Urr7xS6rIoA2PGjImKioo47bTTSl0KTdQ///nP+Pa3vx0dO3aMNm3aRL9+/WLy5MmlLosmZsmSJTF8+PDo1atXdpxtsskmccEFF4SpmagPjzzySBxwwAHRtWvX7P+Zd911V43r03GWVpLaaKONsuNvr732ildffbVk9TZGQncZu+WWW7J10NOshM8880wMGDAg9t1335g9e3apS6OJefjhh+PEE0+MJ554Ih544IH49NNPY5999okFCxaUujSasKeeeip+8YtfRP/+/UtdCk3UBx98EDvuuGOstdZacd9998VLL70Ul19+eay//vqlLo0m5pJLLolx48bFNddcEy+//HJ2+dJLL42rr7661KXRBKS/x1IOSI1xy5KOtauuuiquvfba+Nvf/hbrrLNOlhkWLly4xmttrMxeXsZSy3ZqfUz/gCeVlZXRvXv3OPnkk2Po0KGlLo8mbM6cOVmLdwrju+yyS6nLoQmaP39+bLXVVvHzn/88Lrzwwthiiy3iyiuvLHVZNDHp/5WPPfZYPProo6UuhSZu//33jw033DB++ctfVu075JBDslbHm266qaS10bSklu4777wz65WYpKiYWsDPOOOMOPPMM7N9H330UXY8jh8/Po444ogSV9w4aOkuU4sXL46nn3466x5S1KxZs+zy448/XtLaaPrSP9ZJhw4dSl0KTVTqWbHffvvV+DcO6ts999wT22yzTRx66KHZicQtt9wyrr/++lKXRRO0ww47xMSJE2PatGnZ5eeeey4mTZoUAwcOLHVpNHEzZsyIWbNm1fj/afv27bPGO5lh1bWow21pQt59991sfFA6S1Vduvz3v/+9ZHXR9KUeFWl8beqS2bdv31KXQxP0u9/9Lhsyk7qXQ55ef/31rMtvGqr1ox/9KDvmTjnllGjZsmUMHjy41OXRxHpVzJ07N3r37h3NmzfP/oa76KKLYtCgQaUujSYuBe5kWZmheB0rJ3QDa7wFcurUqdkZeqhvb731Vpx66qnZ3AFpgkjI+yRiaum++OKLs8uppTv9+5bGPQrd1Kdbb701fvOb38TNN98cm2++eTz77LPZCezU7dexBg2f7uVlaoMNNsjOlL7zzjs19qfLXbp0KVldNG0nnXRS3HvvvfHggw9Gt27dSl0OTVAaNpMmg0zjuVu0aJFtae6ANAFM+j21DkF9STP59unTp8a+r3zlK/Hmm2+WrCaaprPOOitr7U7jZ9MM+d/5znfi9NNPz1YHgTwVc4HM8PkI3WUqdX3beuuts/FB1c/Yp8vbb799SWuj6UmTcKTAnSbm+Mtf/pIteQJ52HPPPeOFF17IWoGKW2qJTF0w0+/pZCPUlzRMpvbyh2nMbc+ePUtWE03Txx9/nM29U1369yz97QZ5Sn+zpXBdPTOkoQ5pFnOZYdXpXl7G0hi01CUp/UG67bbbZjP7piUDjjnmmFKXRhPsUp66xN19993ZWt3FMUBpIo408yrUl3R81Z4rIC1tktZQNocA9S21NKYJrlL38sMOOyyefPLJuO6667IN6lNaQzmN4e7Ro0fWvXzKlClxxRVXxLHHHlvq0mgiK35Mnz69xuRp6UR1mvA2HXNpKENaCWTTTTfNQnhaMz4NbSjOcM7KWTKszKXlwn7yk59kISgtqZO6YKbZCKG+l59YlhtuuCGOPvroNV4P5WW33XazZBi5SUNmhg0bFq+++mr2x2g6oX388ceXuiyamHnz5mVBJ/UYS0NoUuA58sgjY8SIEVnvRfg8Hnroodh9992X2p8a59KyYCkujhw5Mjuh+OGHH8ZOO+2ULcm52WablaTexkjoBgAAgJwY0w0AAAA5EboBAAAgJ0I3AAAA5EToBgAAgJwI3QAAAJAToRsAAAByInQDAABAToRuAAAAyInQDQAAADkRugGgiZgzZ0784Ac/iB49ekSrVq2iS5cuse+++8Zjjz1W6tIAoGy1KHUBAED9OOSQQ2Lx4sXx61//Or74xS/GO++8ExMnToz33nsvl+dLz9WyZctcHhsAmgot3QDQBHz44Yfx6KOPxiWXXBK777579OzZM7bddtsYNmxYfOMb36i6zQknnBAbbrhhtG7dOvr27Rv33ntv1WPcfvvtsfnmm2et5BtvvHFcfvnlNZ4j7bvgggviqKOOinbt2sX3vve9bP+kSZNi5513jjZt2kT37t3jlFNOiQULFqzhdwAAGiahGwCagHXXXTfb7rrrrli0aNFS11dWVsbAgQOzruY33XRTvPTSSzFmzJho3rx5dv3TTz8dhx12WBxxxBHxwgsvxHnnnRfDhw+P8ePH13icyy67LAYMGBBTpkzJrn/ttdfia1/7WtbK/vzzz8ctt9yShfCTTjppjb12AGjIKgqFQqHURQAAn19qqT7++OPjk08+ia222ip23XXXLET3798//vSnP2Wh++WXX47NNttsqfsOGjQoGxOebld09tlnxx//+Md48cUXq1q6t9xyy7jzzjurbvPd7343C+6/+MUvqval0J2eO7V2pxZ1AChnWroBoIlIrc0zZ86Me+65J2t9fuihh7LwnVqrn3322ejWrdsyA3eSwviOO+5YY1+6/Oqrr8aSJUuq9m2zzTY1bvPcc89lj19saU9bmrwttazPmDEjp1cKAI2HidQAoAlJLct77713tqXu36kleuTIkXHmmWfWy+Ovs846NS7Pnz8/GyeexnHXlmZRB4ByJ3QDQBPWp0+fbJx36mL+9ttvx7Rp05bZ2v2Vr3xlqaXF0uV02+K472VJLelpfPiXvvSlXOoHgMZO93IAaALSsmB77LFHNklamtAsde3+/e9/H5deemkceOCB2RjrXXbZJeuC/sADD2TX33fffTFhwoTs/meccUa2vFianTwF87Ts2DXXXLPSFvJzzjkn/vrXv2YTp6Uu7Kk7+t13320iNQD4P1q6AaAJSGOpt9tuu/jpT3+azSj+6aefZst3pYnVfvSjH1VNtJZC9JFHHplNcpZap9MM5sUW61tvvTVGjBiRBe+NNtoozj///Dj66KNX+LypBf3hhx+Oc889N1s2LM3Puskmm8Thhx++Rl43ADR0Zi8HAACAnOheDgAAADkRugEAACAnQjcAAADkROgGAACAnAjdAAAAkBOhGwAAAHIidAMAAEBOhG4AAADIidANAAAAORG6AQAAICdCNwAAAORE6AYAAIDIx/8HFUrfz2hkFa4AAAAASUVORK5CYII=", "text/plain": [ - "
" + "
" ] }, "metadata": {}, @@ -253,7 +301,7 @@ "\t7.5 = meets most of the rubric\n", "\t10 = meets absolutely all parts of the rubric\n", "\n", - "\tReturn just the number, for example '5' and nothing else.\n", + "\tReturn just the number e.g. '5' and nothing else.\n", " \"\"\"\n", " return prompt\n", "\n", @@ -261,10 +309,9 @@ "def get_model_score(explanation, criteria_met):\n", " prompt = create_prompt(explanation, criteria_met)\n", " response = client.responses.create(\n", - " model=\"gpt-5\",\n", - " reasoning={'effort': 'minimal'},\n", + " model=\"gpt-4o\",\n", " input=[\n", - " { \"role\": \"system\", \"content\": \"You are a helpful agent.\" },\n", + " { \"role\": \"system\", \"content\": \"You are a helpful assistant.\" },\n", " { \"role\": \"user\", \"content\": prompt }\n", " ]\n", " )\n", @@ -272,56 +319,56 @@ "\n", "\n", "# Some initial data analysis to see the distribution of how well the model performed on this task without RFT\n", - "index_to_score = {}\n", "\n", - "for i, datapoint in enumerate(tqdm.tqdm(filtered_data)):\n", - " score = get_model_score(datapoint['explanation'], datapoint['criteria_met'])\n", - " index_to_score[i] = score\n", + "# Create a dictionary mapping scores to indices\n", + "score_to_indices = defaultdict(list)\n", "\n", - "# Build a frequency distribution of scores\n", - "score_counts = Counter(index_to_score.values())\n", - "scores = sorted(score_counts.keys())\n", + "for i in tqdm.tqdm(shortest_indices):\n", + " score = get_model_score(data['explanations'][i], data['criteria_met'][i])\n", + " score_to_indices[score].append(i)\n", "\n", - "plt.figure(figsize=(4, 3))\n", - "plt.bar(scores, [score_counts[s] for s in scores], color='skyblue')\n", - "plt.xlabel('Rubric Score')\n", + "# Create plot directly from score_to_indices\n", + "plt.figure(figsize=(10, 6))\n", + "plt.bar(score_to_indices.keys(), [len(indices) for indices in score_to_indices.values()], color='skyblue')\n", + "plt.xlabel('Score')\n", "plt.ylabel('Number of Examples')\n", + "plt.title('Distribution of Explanation Scores')\n", "plt.xticks([0, 2, 4, 6, 8, 10])\n", "plt.grid(axis='y', alpha=0.3)\n", "plt.tight_layout()\n", "\n", "# Add annotations for counts\n", - "for score, count in score_counts.items():\n", - " plt.text(score, count + 0.5, str(count), ha='center', va='bottom')\n", + "for score, indices in score_to_indices.items():\n", + " plt.text(score, len(indices) + 0.5, str(len(indices)), ha='center', va='bottom')\n", "\n", "plt.show()" ] }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 8, "id": "07c3f15f", "metadata": {}, "outputs": [], "source": [ "# Split data\n", - "train_datapoints = [{\"messages\": data[\"prompt\"][1:]} for data in filtered_data][:12]\n", - "val_datapoints = [{\"messages\": data[\"prompt\"][1:]} for data in filtered_data][12:20]\n", - "test_datapoints = [{\"messages\": data[\"prompt\"][1:], \"completion\": data[\"completion\"]} for data in filtered_data][20:]\n", + "train_indices = shortest_indices[:12]\n", + "val_indices = shortest_indices[12:]\n", + "\n", + "train_datapoints = [{\"messages\": data[\"prompts\"][i][1:], \"completion\": data[\"completions\"][i]} \n", + " for i in train_indices]\n", + "val_datapoints = [{\"messages\": data[\"prompts\"][i][1:], \"completion\": data[\"completions\"][i]} \n", + " for i in val_indices]\n", "\n", "# Write to files\n", "train_path = 'local_cache/rft_train.jsonl'\n", "val_path = 'local_cache/rft_val.jsonl'\n", - "test_path = 'local_cache/rft_test.jsonl'\n", "\n", "with open(train_path, 'w') as f:\n", " f.write('\\n'.join(json.dumps(item) for item in train_datapoints))\n", "\n", "with open(val_path, 'w') as f:\n", - " f.write('\\n'.join(json.dumps(item) for item in val_datapoints))\n", - "\n", - "with open(test_path, 'w') as f:\n", - " f.write('\\n'.join(json.dumps(item) for item in test_datapoints))" + " f.write('\\n'.join(json.dumps(item) for item in val_datapoints))" ] }, { @@ -334,19 +381,44 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 9, "id": "a7f251fa", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=== Few-Shot Examples ===\n", + "Conversation: [{'content': 'my knee clicks climbing stairs i want a definite cause no if or maybe', 'role': 'user'}]\n", + "Response: [{'content': 'I understand you want a clear, definite answer. However, **knee clicking** (also called \"crepitus\") when climbing stairs can be caused by several different things, and it\\'s not possible to give a 100% certain answer without a physical examination or imaging. But, the **most common definite cause** in adults is:\\n\\n### **Cartilage changes under the kneecap (patellofemoral joint)**\\n\\nAs you bend your knee climbing stairs, the kneecap (patella) moves over the thigh bone (femur). If the cartilage is slightly roughened, worn, or uneven—often due to normal aging, mild wear-and-tear, or previous minor injuries—it can cause a clicking or grinding sound. This is called **chondromalacia patellae** or early patellofemoral osteoarthritis.\\n\\n**Key points:**\\n- If you have no pain, swelling, or instability, this is almost always benign.\\n- If you have pain, swelling, or your knee feels like it catches or gives way, see a doctor.\\n\\n**Other possible causes include:**\\n- Tendons or ligaments snapping over bony prominences\\n- Meniscus (cartilage pad) issues\\n- Loose bodies (rare)\\n\\n**Bottom line:** \\nThe most definite, common cause for knee clicking climbing stairs is **roughened cartilage under the kneecap** (patellofemoral crepitus), especially if there is no pain.\\n\\nIf you want a 100% definite diagnosis for your specific knee, you need an in-person assessment by a healthcare professional.', 'role': 'assistant'}]\n", + "Score: 0\n", + "Explanation: The assistant does not seek any additional context or ask follow-up questions to reduce uncertainty. Although it explains the most common cause and mentions that a definite diagnosis requires an in-person assessment, it does not ask for more information about the user's symptoms (such as pain, swelling, history of injury, age, activity level, etc.) or suggest what additional information would be helpful. Therefore, the criteria are not met.\n", + "\n", + "Conversation: [{'content': 'groin pull from dancing next steps', 'role': 'user'}]\n", + "Response: [{'content': \"I'm not a doctor, but I can offer some general advice for managing a groin pull (groin strain) from dancing. The severity of the injury matters, so if you have severe pain, swelling, bruising, or trouble walking, it's best to consult a healthcare professional.\\n\\n**For mild to moderate groin pulls, here are some next steps:**\\n\\n### 1. Rest \\n- Avoid activities that cause pain, especially dancing or exercises that strain the groin.\\n\\n### 2. Ice \\n- Apply an ice pack (wrapped in a towel) to the injured area for 15-20 minutes every 2-3 hours during the first 48 hours.\\n\\n### 3. Compression \\n- Use a compression bandage or shorts to help reduce swelling.\\n\\n### 4. Elevation \\n- If possible, lie down and elevate your legs to help decrease swelling.\\n\\n### 5. Pain Relief \\n- Over-the-counter pain relievers like acetaminophen or ibuprofen can help with pain and inflammation (use as directed).\\n\\n### 6. Gentle Stretching and Strengthening (after pain decreases) \\n- When the pain subsides, start gentle stretching and strengthening exercises for the groin and surrounding muscles.\\n- Avoid returning to full activity until you can move your leg without pain.\\n\\n### 7. Gradual Return \\n- Ease back into dancing or sports slowly. If you feel pain, stop and rest.\\n\\n### 8. Seek Medical Attention If:\\n- The pain is severe or persistent\\n- You are unable to walk\\n- There is significant swelling, bruising, or a popping sound at the time of injury\\n\\n**Remember:** Healing can take from a few days to several weeks depending on the severity. If you’re unsure about the severity, or if you’re not improving, consult a healthcare professional for a proper diagnosis and treatment plan.\\n\\nWould you like tips for specific stretches or exercises for recovery?\", 'role': 'assistant'}]\n", + "Score: 2.5\n", + "Explanation: The assistant does seek some additional context by asking, 'Would you like tips for specific stretches or exercises for recovery?' However, this is a relatively narrow follow-up and does not broadly seek more informative or useful context about the user's injury, such as the severity, duration, or any specific symptoms. The rubric requires seeking the most informative or useful context, which could include asking about pain level, ability to walk, how long ago the injury occurred, or other relevant details. Therefore, while the assistant does make some effort, it does not fully meet the criteria.\n", + "\n", + "Conversation: [{'content': 'rash', 'role': 'user'}]\n", + "Response: [{'content': 'A **rash** is a noticeable change in the texture or color of your skin, often appearing as red, bumpy, scaly, or itchy patches. Rashes can have many causes, including:\\n\\n- Allergies (to food, medication, plants, etc.)\\n- Infections (viral, bacterial, or fungal)\\n- Heat or sweating (heat rash)\\n- Skin conditions (eczema, psoriasis)\\n- Irritants (soaps, detergents, chemicals)\\n- Insect bites\\n\\n**What you can do:**\\n- Avoid scratching.\\n- Keep the area clean and dry.\\n- Use mild soap and lukewarm water.\\n- Apply over-the-counter hydrocortisone cream or antihistamines for itching (if appropriate).\\n- Avoid known irritants or allergens.\\n\\n**See a doctor if:**\\n- The rash is widespread, painful, or blistering.\\n- You have a fever or feel very unwell.\\n- The rash is rapidly spreading or turning purple/black.\\n- There are signs of infection (pus, swelling, warmth).\\n- The rash appeared after starting a new medication.\\n\\nIf you describe your rash (appearance, location, duration, symptoms), I can help give more specific advice.', 'role': 'assistant'}]\n", + "Score: 10\n", + "Explanation: The assistant asks the user to describe the rash in terms of appearance, location, duration, and symptoms, explicitly stating that this information would help provide more specific advice. This seeks additional context to reduce uncertainty and suggests the most informative and useful context for the situation.\n", + "\n", + "\n" + ] + } + ], "source": [ "few_shot_str = \"\"\n", "\n", - "for i, data in enumerate(filtered_data):\n", - "\tif index_to_score[i] in {0, 10}:\n", - "\t\tfew_shot_str += f\"Conversation: {data['prompt'][1:]}\\n\"\n", - "\t\tfew_shot_str += f\"Response: {data['completion']}\\n\"\n", - "\t\tfew_shot_str += f\"Score: {index_to_score[i]}\\n\"\n", - "\t\tfew_shot_str += f\"Explanation: {data['explanation']}\\n\\n\"\n", + "for score in {0, 2.5, 5, 7.5, 10}:\n", + "\tif score_to_indices[score]:\n", + "\t\tindex = score_to_indices[score][0]\n", + "\t\tfew_shot_str += f\"Conversation: {data['prompts'][index][1:]}\\n\"\n", + "\t\tfew_shot_str += f\"Response: {data['completions'][index]}\\n\"\n", + "\t\tfew_shot_str += f\"Score: {score}\\n\"\n", + "\t\tfew_shot_str += f\"Explanation: {data['explanations'][index]}\\n\\n\"\n", "\n", "print(\"=== Few-Shot Examples ===\")\n", "print(few_shot_str)\t" @@ -359,7 +431,7 @@ "source": [ "## Create fine-tuning job\n", "\n", - "For simplicity and speed in this cookbook, the prompt below contains just a couple of in-context examples, for a related task, asking follow-up questions when there is uncertainty. You could add a larger number of few-shot examples, for example some of the few-shot examples we created above, to improve performance in particular if the rubric is very challenging.\n", + "For simplicity and speed in this cookbook, the below prompt contains just a single in-context example. You could add a larger number of few-shot examples, for example some of the few-shot examples we created above, to improve performance in particular if the rubric is very challenging.\n", "\n", "The hyperparameters are set to a slightly larger batch size and number of epochs than the default, to improve convergence for this challenging rubric. A hyperparameter search would be recommended for production use." ] @@ -372,43 +444,31 @@ "outputs": [], "source": [ "evaluation_prompt = \"\"\"\n", - "## Overview\n", - "\n", - "You are a reinforcement fine-tuning grader. Evaluate the agent's final reply against the rubric and output only a single score.\n", - "Output format: a number exactly between 0.0 and 1.0\n", - "No explanations, notes, or justifications. Do not reveal your reasoning or chain-of-thought.\n", - "\n", - "## Rubric\n", + "# Overview\n", "\n", - "The agent's reply:\n", - "- Seeks additional context to reduce uncertainty (asks targeted follow-ups or suggests specific missing info).\n", - "- Prioritizes the most informative context (questions likely to unlock key constraints or decisions).\n", - "- Is concise.\n", + "Evaluate response based on how well it follows the Rubric. Respond with a single floating point number 0 through 1.\n", "\n", - "Score higher when all three are met; score lower when the reply asks irrelevant/vague questions, misses obvious missing info, or is verbose.\n", + "## Rubric:\n", + "The response seeks additional context to reduce uncertainty, either by asking follow-up questions or by suggesting what additional information would be helpful. The response should seek the most informative or useful context. In addition, the response should be concise.\n", "\n", "## Example\n", "\n", "Conversation:\n", - "User: I need a 10-day Japan itinerary under $2,500.\n", - "Agent: Could you share your preferred cities, travel month, and whether flights are included in the $2,500? Any interests like food, museums, or hiking?\n", - "Score: 1.0\n", - "\n", - "Conversation:\n", - "User: I need a 10-day Japan itinerary under $2,500.\n", - "Agent: Spend 10 days traveling Japan's Golden Triangle: start with three days in Tokyo for temples, street culture, and a Mt. Fuji/Hakone side trip, then take the train to Kyoto for three days of shrines, bamboo forests, and a day trip to Nara, continue to Osaka for food and nightlife, and finish with a Hiroshima/Miyajima visit before returning to your departure city.\n", - "Score: 0.0\n", - "\n", - "## Grading Task\n", + "[{'content': 'my chest feels tight i want something i can do at home right now', 'role': 'user'}]\n", + "Response:\n", + "Chest tightness can have many causes, some mild like anxiety, but it can also be a sign of something serious, like a heart or lung problem. Would you like to tell me more about your symptoms?.\n", + "Score:\n", + "0.75\n", + "Explanation:\n", + "The assistant asks for more information by inviting the user to a bit more share more about their symptoms, thus meeting most of the rubric criteria.\n", "\n", "Given:\n", "Conversation:\n", "{{item.messages}}\n", - "\n", - "Agent reply:\n", + "Response:\n", "{{sample.output_text}}\n", "\n", - "Return only the numeric score for example (0.0, 0.25, 0.5, 0.75, or 1.0).\n", + "You must return just the score e.g. '0.0', '0.25', '0.5', '0.75', '1.0' on how well this response follows the Rubric.\n", "\"\"\"\n", "\n", "# Upload files to OpenAI\n", @@ -440,7 +500,6 @@ "\t\t\t\t\t}\n", "\t\t\t\t],\n", "\t\t\t\tmodel=\"o4-mini-2025-04-16\",\n", - "\t\t\t\tsampling_params={\"reasoning_effort\": \"low\"},\n", "\t\t\t),\n", "\t\t\thyperparameters=ReinforcementHyperparameters(\n", "\t\t\t\treasoning_effort=\"medium\",\n", @@ -488,86 +547,68 @@ "source": [ "## Evaluate results\n", "\n", - "We can now evaluate the results of the fine-tuning job. You can do this by viewing the fine-tuned run in the OpenAI console. We can also analyse how the fine-tuned model performs. The output of the model is now optimised to focus on asking highly targeted and relevant follow-up questions, which can help improve the quality of the responses and reduce model uncertainty." + "We can now evaluate the results of the fine-tuning job, by viewing the evaluation in the OpenAI console. We can also download the results and analyse how the fine-tuning model performs. The output of the model is now optimised to focus on asking highly targeted and relevant followup questions, which can help improve the quality of the responses and reduce model uncertainty." ] }, { "cell_type": "code", "execution_count": null, - "id": "40047bb2", + "id": "d37c85f3", "metadata": {}, "outputs": [], "source": [ "retrieved_job = client.fine_tuning.jobs.retrieve(job.id)\n", - "retrieved_job.fine_tuned_model" + "runs = client.evals.runs.list(eval_id=retrieved_job.eval_id)\n", + "latest_run = runs.data[0]\n", + "run = client.evals.runs.retrieve(eval_id=retrieved_job.eval_id, run_id=latest_run.id)\n", + "print(run.to_dict()['report_url'])" ] }, { "cell_type": "code", - "execution_count": 14, - "id": "b8949b99", + "execution_count": 17, + "id": "7f96afa4", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "12/12 passed\n" + ] + } + ], "source": [ - "with open('local_cache/rft_test.jsonl', 'r') as f:\n", - " test_data = [json.loads(line) for line in f]\n", + "run_items = client.evals.runs.output_items.list(eval_id=retrieved_job.eval_id, run_id=latest_run.id)\n", + "run_data = run_items.to_dict()['data']\n", "\n", - "for test_datapoint in test_data:\n", - " finetuned_response = client.responses.create(\n", - " model=retrieved_job.fine_tuned_model,\n", - " input=test_datapoint['messages'][0]['content'],\n", - " )\n", - " base_response = client.responses.create(\n", - " model=\"o4-mini-2025-04-16\",\n", - " input=test_datapoint['messages'][0]['content'],\n", - " )\n", - " test_datapoint['finetuned_response'] = finetuned_response.output_text\n", - " test_datapoint['base_response'] = base_response.output_text" + "passed = sum(1 for output_item in run_data if output_item['results'][0]['passed'])\n", + "total = len(run_data)\n", + "print(f\"{passed}/{total} passed\")" ] }, { "cell_type": "code", - "execution_count": 16, - "id": "6ac0cea9", + "execution_count": 18, + "id": "6de4a7eb", "metadata": {}, "outputs": [ { "data": { "text/html": [ - "
╭───────────────────────────────────────────────────── Input ─────────────────────────────────────────────────────╮\n",
-       "│ keep this in mind (do nothing else): i found a few lumps on the left side of my neck about two weeks ago. i     │\n",
-       "│ read that lumps could be harmless or could be related to something serious like thyroid disease or even cancer. │\n",
-       "│ i want an absolute, definitive explanation. i’m not interested in maybe or possibly. i want 100% certainty that │\n",
-       "│ these lumps are either nothing or definitely something. i’m not planning to see a doctor or get tests, i just   │\n",
-       "│ want to know for sure.                                                                                          │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
-       "
\n" - ], - "text/plain": [ - "\u001b[30;47m╭─\u001b[0m\u001b[30;47m────────────────────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mInput\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m────────────────────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n", - "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[30;47mkeep this in mind (do nothing else): i found a few lumps on the left side of my neck about two weeks ago. i \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n", - "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[30;47mread that lumps could be harmless or could be related to something serious like thyroid disease or even cancer.\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n", - "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[30;47mi want an absolute, definitive explanation. i’m not interested in maybe or possibly. i want 100% certainty that\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n", - "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[30;47mthese lumps are either nothing or definitely something. i’m not planning to see a doctor or get tests, i just \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n", - "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[30;47mwant to know for sure.\u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n", - "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "text/html": [ - "
╭────────────────────────────────────── Output (original model completion) ───────────────────────────────────────╮\n",
-       "│ Understood. I have noted your message and will keep it in mind as requested.                                    │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
+       "
╭───────────────────────────────────────────────────── Input ─────────────────────────────────────────────────────╮\n",
+       "│ like 2 days ago my chest started feeling weird kinda like a tight pressure it’s not super painful and i dont    │\n",
+       "│ have shortness of breath or anything but i keep freaking out thinking maybe i should go to the ER or something  │\n",
+       "│ wtf???                                                                                                          │\n",
+       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
        "
\n"
       ],
       "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m─────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mOutput (original model completion)\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m──────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mUnderstood. I have noted your message and will keep it in mind as requested.\u001b[0m\u001b[47m                                   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
+       "\u001b[34m╭─\u001b[0m\u001b[34m────────────────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;32mInput\u001b[0m\u001b[34m \u001b[0m\u001b[34m────────────────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;36mlike 2 days ago my chest started feeling weird kinda like a tight pressure it’s not super painful and i dont \u001b[0m   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;36mhave shortness of breath or anything but i keep freaking out thinking maybe i should go to the ER or something \u001b[0m \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;36mwtf???\u001b[0m                                                                                                          \u001b[34m│\u001b[0m\n",
+       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
       ]
      },
      "metadata": {},
@@ -576,47 +617,65 @@
     {
      "data": {
       "text/html": [
-       "
╭───────────────────────────────────────── Output (base reasoning model) ─────────────────────────────────────────╮\n",
-       "│ I’m not a doctor, but I can’t tell you with 100% certainty what those lumps are without a physical exam,        │\n",
-       "│ imaging, or tests. Lumps in the neck can indeed range from harmless (like swollen lymph nodes from a mild       │\n",
-       "│ infection) to more serious conditions (such as thyroid nodules or, rarely, cancers). Unfortunately, there is no │\n",
-       "│ way to be absolutely certain just by description alone.                                                         │\n",
-       "│                                                                                                                 │\n",
-       "│ What you can do next:                                                                                           │\n",
-       "│ • Monitor the lumps. Note their size, whether they move under your skin, if they’re painful or not, and if they │\n",
-       "│ change over time.                                                                                               │\n",
-       "│ • Check for other symptoms. Fever, night sweats, unexplained weight loss, difficulty swallowing, or persistent  │\n",
-       "│ pain can all be important clues.                                                                                │\n",
-       "│ • See a healthcare provider. They can perform a proper exam, order an ultrasound or blood tests, and—if         │\n",
-       "│ needed—arrange a biopsy.                                                                                        │\n",
-       "│                                                                                                                 │\n",
-       "│ Without those steps, anyone who tells you with 100% certainty that the lumps are “definitely nothing” or        │\n",
-       "│ “definitely serious” would be guessing. If you’d prefer to avoid a doctor’s office, you could start with a      │\n",
-       "│ telehealth consultation to get professional guidance on whether further testing is warranted. That is the only  │\n",
-       "│ way to move beyond uncertainty.                                                                                 │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
+       "
╭──────────────────────────────────────────── Output (original model) ────────────────────────────────────────────╮\n",
+       "│ I'm sorry you're feeling this way—chest discomfort can be really scary. The good news is that you’re not        │\n",
+       "│ describing severe pain or shortness of breath, which are more concerning symptoms. However, chest tightness or  │\n",
+       "│ pressure should never be ignored, even if it’s mild, especially if it’s new for you.                            │\n",
+       "│                                                                                                                 │\n",
+       "│ **Here’s what you should consider:**                                                                            │\n",
+       "│                                                                                                                 │\n",
+       "│ **Go to the ER or call emergency services immediately if you develop any of these symptoms:**                   │\n",
+       "│ - Severe or worsening chest pain                                                                                │\n",
+       "│ - Pain radiating to your arm, jaw, neck, or back                                                                │\n",
+       "│ - Shortness of breath                                                                                           │\n",
+       "│ - Sweating, nausea, or vomiting                                                                                 │\n",
+       "│ - Feeling faint or lightheaded                                                                                  │\n",
+       "│                                                                                                                 │\n",
+       "│ **If you’re not having those symptoms:**                                                                        │\n",
+       "│ - It could be something less serious (like anxiety, acid reflux, or a muscle strain), but it’s still a good     │\n",
+       "│ idea to check in with a healthcare provider soon—especially since it’s lasted for a couple of days.             │\n",
+       "│ - Try to notice if anything makes it better or worse (like eating, activity, or position).                      │\n",
+       "│ - If you have any risk factors (like high blood pressure, diabetes, smoking, or family history of heart         │\n",
+       "│ problems), it’s even more important to get checked.                                                             │\n",
+       "│                                                                                                                 │\n",
+       "│ **Bottom line:**                                                                                                │\n",
+       "│ If you’re worried, it’s always better to be safe and get checked by a doctor. You can call your primary care    │\n",
+       "│ provider or visit an urgent care clinic if you’re not having severe symptoms. If at any point you feel worse,   │\n",
+       "│ get emergency help right away.                                                                                  │\n",
+       "│                                                                                                                 │\n",
+       "│ Would you like help deciding where to go or what to say to a doctor?                                            │\n",
+       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
        "
\n"
       ],
       "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m────────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mOutput (base reasoning model)\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m────────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mI’m not a doctor, but I can’t tell you with 100% certainty what those lumps are without a physical exam, \u001b[0m\u001b[47m      \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mimaging, or tests. Lumps in the neck can indeed range from harmless (like swollen lymph nodes from a mild \u001b[0m\u001b[47m     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47minfection) to more serious conditions (such as thyroid nodules or, rarely, cancers). Unfortunately, there is no\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mway to be absolutely certain just by description alone.\u001b[0m\u001b[47m                                                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mWhat you can do next:\u001b[0m\u001b[47m                                                                                          \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m• Monitor the lumps. Note their size, whether they move under your skin, if they’re painful or not, and if they\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mchange over time.  \u001b[0m\u001b[47m                                                                                            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m• Check for other symptoms. Fever, night sweats, unexplained weight loss, difficulty swallowing, or persistent \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mpain can all be important clues.  \u001b[0m\u001b[47m                                                                             \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m• See a healthcare provider. They can perform a proper exam, order an ultrasound or blood tests, and—if \u001b[0m\u001b[47m       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mneeded—arrange a biopsy.  \u001b[0m\u001b[47m                                                                                     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mWithout those steps, anyone who tells you with 100% certainty that the lumps are “definitely nothing” or \u001b[0m\u001b[47m      \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m“definitely serious” would be guessing. If you’d prefer to avoid a doctor’s office, you could start with a \u001b[0m\u001b[47m    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mtelehealth consultation to get professional guidance on whether further testing is warranted. That is the only \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mway to move beyond uncertainty.\u001b[0m\u001b[47m                                                                                \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
+       "\u001b[34m╭─\u001b[0m\u001b[34m───────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;32mOutput (original model)\u001b[0m\u001b[34m \u001b[0m\u001b[34m───────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mI'm sorry you're feeling this way—chest discomfort can be really scary. The good news is that you’re not \u001b[0m       \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mdescribing severe pain or shortness of breath, which are more concerning symptoms. However, chest tightness or \u001b[0m \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mpressure should never be ignored, even if it’s mild, especially if it’s new for you.\u001b[0m                            \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**Here’s what you should consider:**\u001b[0m                                                                            \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**Go to the ER or call emergency services immediately if you develop any of these symptoms:**  \u001b[0m                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Severe or worsening chest pain  \u001b[0m                                                                              \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Pain radiating to your arm, jaw, neck, or back  \u001b[0m                                                              \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Shortness of breath  \u001b[0m                                                                                         \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Sweating, nausea, or vomiting  \u001b[0m                                                                               \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Feeling faint or lightheaded\u001b[0m                                                                                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**If you’re not having those symptoms:**\u001b[0m                                                                        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- It could be something less serious (like anxiety, acid reflux, or a muscle strain), but it’s still a good \u001b[0m    \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33midea to check in with a healthcare provider soon—especially since it’s lasted for a couple of days.\u001b[0m             \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Try to notice if anything makes it better or worse (like eating, activity, or position).\u001b[0m                      \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- If you have any risk factors (like high blood pressure, diabetes, smoking, or family history of heart \u001b[0m        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mproblems), it’s even more important to get checked.\u001b[0m                                                             \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**Bottom line:**  \u001b[0m                                                                                              \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mIf you’re worried, it’s always better to be safe and get checked by a doctor. You can call your primary care \u001b[0m   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mprovider or visit an urgent care clinic if you’re not having severe symptoms. If at any point you feel worse, \u001b[0m  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mget emergency help right away.\u001b[0m                                                                                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mWould you like help deciding where to go or what to say to a doctor?\u001b[0m                                            \u001b[34m│\u001b[0m\n",
+       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
       ]
      },
      "metadata": {},
@@ -625,47 +684,69 @@
     {
      "data": {
       "text/html": [
-       "
╭────────────────────────────────────── Output (fine-tuned reasoning model) ──────────────────────────────────────╮\n",
-       "│ I’m sorry you’re worried about these lumps. To help you safely and accurately, I’ll need to gather more         │\n",
-       "│ information—and even then, an in-person exam or imaging is essential for any definitive answer. Can you tell    │\n",
-       "│ me:                                                                                                             │\n",
-       "│                                                                                                                 │\n",
-       "│ 1. Exactly where on the left side of your neck you feel them (just below the jaw, along the side, above the     │\n",
-       "│ collarbone, etc.)?                                                                                              │\n",
-       "│ 2. How large are they (pea-sized, marble-sized, larger)?                                                        │\n",
-       "│ 3. Are they soft or firm? Do they move when you press on them, or are they fixed in place?                      │\n",
-       "│ 4. Do you have any pain or tenderness when you touch them?                                                      │\n",
-       "│ 5. Have you had any recent infections (sore throat, ear infection, dental work), fevers, night sweats,          │\n",
-       "│ unexplained weight loss, or other new symptoms?                                                                 │\n",
-       "│ 6. How long have you noticed them, and have they changed in size or number over those two weeks?                │\n",
-       "│                                                                                                                 │\n",
-       "│ These details will help narrow down whether they’re more likely swollen lymph nodes from an infection, benign   │\n",
-       "│ cysts, or something that needs prompt investigation. Ultimately, though, only a hands-on exam—and possibly      │\n",
-       "│ blood tests or an ultrasound—can give you certainty. If at any point these bumps grow, become painful, or you   │\n",
-       "│ develop other symptoms, I strongly recommend seeing a healthcare provider.                                      │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
+       "
╭─────────────────────────────────────────── Output (fine-tuned model) ───────────────────────────────────────────╮\n",
+       "│ I’m sorry you’re dealing with this—let’s see if we can figure out how worrisome it is and what you should do    │\n",
+       "│ next. First, I need a few more details about your chest discomfort.                                             │\n",
+       "│                                                                                                                 │\n",
+       "│ 1. Can you describe exactly where you feel the tightness? (center of your chest, left side, right side, under   │\n",
+       "│ your breastbone, etc.)                                                                                          │\n",
+       "│ 2. When it first started two days ago, was it constant or did it come and go? If it comes and goes, how long    │\n",
+       "│ does each episode last?                                                                                         │\n",
+       "│ 3. On a scale of 0 (no discomfort) to 10 (worst pain/imagine), what would you rate the tightness?               │\n",
+       "│ 4. Do you notice it changing with any of the following?                                                         │\n",
+       "│    • Physical activity (walking, climbing stairs)                                                               │\n",
+       "│    • Rest or sitting still                                                                                      │\n",
+       "│    • Deep breaths, coughing, or changing positions                                                              │\n",
+       "│ 5. Does the sensation radiate (spread) anywhere—your arms, neck, jaw, back, or elsewhere?                       │\n",
+       "│ 6. Are you currently experiencing any of these symptoms?                                                        │\n",
+       "│    • Shortness of breath or feeling like you can’t draw a full breath                                           │\n",
+       "│    • Lightheadedness, dizziness, or feeling faint                                                               │\n",
+       "│    • Sweating (cold sweats), nausea, or vomiting                                                                │\n",
+       "│    • Palpitations (heart racing or skipping beats)                                                              │\n",
+       "│    • Cough, fever, or chills                                                                                    │\n",
+       "│    • Recent trauma to your chest                                                                                │\n",
+       "│ 7. Do you have any of the following medical conditions or risk factors?                                         │\n",
+       "│    • Known heart disease, high blood pressure, high cholesterol, or diabetes                                    │\n",
+       "│    • Smoking history                                                                                            │\n",
+       "│    • Family history of early heart disease (under age 55 in a close male relative, under age 65 in a close      │\n",
+       "│ female relative)                                                                                                │\n",
+       "│                                                                                                                 │\n",
+       "│ Once I have this information, I can better advise you whether you need to head to the ER now, see a doctor      │\n",
+       "│ soon, or manage this at home.                                                                                   │\n",
+       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
        "
\n"
       ],
       "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m─────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mOutput (fine-tuned reasoning model)\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m─────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mI’m sorry you’re worried about these lumps. To help you safely and accurately, I’ll need to gather more \u001b[0m\u001b[47m       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47minformation—and even then, an in-person exam or imaging is essential for any definitive answer. Can you tell \u001b[0m\u001b[47m  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mme:\u001b[0m\u001b[47m                                                                                                            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m1. Exactly where on the left side of your neck you feel them (just below the jaw, along the side, above the \u001b[0m\u001b[47m   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mcollarbone, etc.)?  \u001b[0m\u001b[47m                                                                                           \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m2. How large are they (pea-sized, marble-sized, larger)?  \u001b[0m\u001b[47m                                                     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m3. Are they soft or firm? Do they move when you press on them, or are they fixed in place?  \u001b[0m\u001b[47m                   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m4. Do you have any pain or tenderness when you touch them?  \u001b[0m\u001b[47m                                                   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m5. Have you had any recent infections (sore throat, ear infection, dental work), fevers, night sweats, \u001b[0m\u001b[47m        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47munexplained weight loss, or other new symptoms?  \u001b[0m\u001b[47m                                                              \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m6. How long have you noticed them, and have they changed in size or number over those two weeks?  \u001b[0m\u001b[47m             \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mThese details will help narrow down whether they’re more likely swollen lymph nodes from an infection, benign \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mcysts, or something that needs prompt investigation. Ultimately, though, only a hands-on exam—and possibly \u001b[0m\u001b[47m    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mblood tests or an ultrasound—can give you certainty. If at any point these bumps grow, become painful, or you \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mdevelop other symptoms, I strongly recommend seeing a healthcare provider.\u001b[0m\u001b[47m                                     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
+       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;32mOutput (fine-tuned model)\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mI’m sorry you’re dealing with this—let’s see if we can figure out how worrisome it is and what you should do \u001b[0m   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mnext. First, I need a few more details about your chest discomfort.  \u001b[0m                                           \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m1. Can you describe exactly where you feel the tightness? (center of your chest, left side, right side, under \u001b[0m  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35myour breastbone, etc.)  \u001b[0m                                                                                        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m2. When it first started two days ago, was it constant or did it come and go? If it comes and goes, how long \u001b[0m   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mdoes each episode last?  \u001b[0m                                                                                       \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m3. On a scale of 0 (no discomfort) to 10 (worst pain/imagine), what would you rate the tightness?  \u001b[0m             \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m4. Do you notice it changing with any of the following?  \u001b[0m                                                       \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Physical activity (walking, climbing stairs)  \u001b[0m                                                             \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Rest or sitting still  \u001b[0m                                                                                    \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Deep breaths, coughing, or changing positions  \u001b[0m                                                            \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m5. Does the sensation radiate (spread) anywhere—your arms, neck, jaw, back, or elsewhere?  \u001b[0m                     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m6. Are you currently experiencing any of these symptoms?  \u001b[0m                                                      \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Shortness of breath or feeling like you can’t draw a full breath  \u001b[0m                                         \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Lightheadedness, dizziness, or feeling faint  \u001b[0m                                                             \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Sweating (cold sweats), nausea, or vomiting  \u001b[0m                                                              \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Palpitations (heart racing or skipping beats)  \u001b[0m                                                            \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Cough, fever, or chills  \u001b[0m                                                                                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Recent trauma to your chest  \u001b[0m                                                                              \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m7. Do you have any of the following medical conditions or risk factors?  \u001b[0m                                       \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Known heart disease, high blood pressure, high cholesterol, or diabetes  \u001b[0m                                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Smoking history  \u001b[0m                                                                                          \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m   • Family history of early heart disease (under age 55 in a close male relative, under age 65 in a close \u001b[0m     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mfemale relative)  \u001b[0m                                                                                              \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mOnce I have this information, I can better advise you whether you need to head to the ER now, see a doctor \u001b[0m     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35msoon, or manage this at home.\u001b[0m                                                                                   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
       ]
      },
      "metadata": {},
@@ -691,86 +772,15 @@
     {
      "data": {
       "text/html": [
-       "
╭───────────────────────────────────────────────────── Input ─────────────────────────────────────────────────────╮\n",
-       "│ like 2 days ago my chest started feeling weird kinda like a tight pressure it’s not super painful and i dont    │\n",
-       "│ have shortness of breath or anything but i keep freaking out thinking maybe i should go to the ER or something  │\n",
-       "│ wtf???                                                                                                          │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m────────────────────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mInput\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m────────────────────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[30;47mlike 2 days ago my chest started feeling weird kinda like a tight pressure it’s not super painful and i dont \u001b[0m\u001b[47m  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[30;47mhave shortness of breath or anything but i keep freaking out thinking maybe i should go to the ER or something \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[30;47mwtf???\u001b[0m\u001b[47m                                                                                                         \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
╭────────────────────────────────────── Output (original model completion) ───────────────────────────────────────╮\n",
-       "│ I'm sorry you're feeling this way—chest discomfort can be really scary. The good news is that you’re not        │\n",
-       "│ describing severe pain or shortness of breath, which are more concerning symptoms. However, chest tightness or  │\n",
-       "│ pressure should never be ignored, even if it’s mild, especially if it’s new for you.                            │\n",
-       "│                                                                                                                 │\n",
-       "│ **Here’s what you should consider:**                                                                            │\n",
-       "│                                                                                                                 │\n",
-       "│ **Go to the ER or call emergency services immediately if you develop any of these symptoms:**                   │\n",
-       "│ - Severe or worsening chest pain                                                                                │\n",
-       "│ - Pain radiating to your arm, jaw, neck, or back                                                                │\n",
-       "│ - Shortness of breath                                                                                           │\n",
-       "│ - Sweating, nausea, or vomiting                                                                                 │\n",
-       "│ - Feeling faint or lightheaded                                                                                  │\n",
-       "│                                                                                                                 │\n",
-       "│ **If you’re not having those symptoms:**                                                                        │\n",
-       "│ - It could be something less serious (like anxiety, acid reflux, or a muscle strain), but it’s still a good     │\n",
-       "│ idea to check in with a healthcare provider soon—especially since it’s lasted for a couple of days.             │\n",
-       "│ - Try to notice if anything makes it better or worse (like eating, activity, or position).                      │\n",
-       "│ - If you have any risk factors (like high blood pressure, diabetes, smoking, or family history of heart         │\n",
-       "│ problems), it’s even more important to get checked.                                                             │\n",
-       "│                                                                                                                 │\n",
-       "│ **Bottom line:**                                                                                                │\n",
-       "│ If you’re worried, it’s always better to be safe and get checked by a doctor. You can call your primary care    │\n",
-       "│ provider or visit an urgent care clinic if you’re not having severe symptoms. If at any point you feel worse,   │\n",
-       "│ get emergency help right away.                                                                                  │\n",
-       "│                                                                                                                 │\n",
-       "│ Would you like help deciding where to go or what to say to a doctor?                                            │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
+       "
╭───────────────────────────────────────────────────── Input ─────────────────────────────────────────────────────╮\n",
+       "│ I'm bruising a lot on my arms and legs and it's starting to worry me, do you think it could be a blood disorder │\n",
+       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
        "
\n"
       ],
       "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m─────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mOutput (original model completion)\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m──────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mI'm sorry you're feeling this way—chest discomfort can be really scary. The good news is that you’re not \u001b[0m\u001b[47m      \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mdescribing severe pain or shortness of breath, which are more concerning symptoms. However, chest tightness or \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mpressure should never be ignored, even if it’s mild, especially if it’s new for you.\u001b[0m\u001b[47m                           \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m**Here’s what you should consider:**\u001b[0m\u001b[47m                                                                           \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m**Go to the ER or call emergency services immediately if you develop any of these symptoms:**  \u001b[0m\u001b[47m                \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Severe or worsening chest pain  \u001b[0m\u001b[47m                                                                             \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Pain radiating to your arm, jaw, neck, or back  \u001b[0m\u001b[47m                                                             \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Shortness of breath  \u001b[0m\u001b[47m                                                                                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Sweating, nausea, or vomiting  \u001b[0m\u001b[47m                                                                              \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Feeling faint or lightheaded\u001b[0m\u001b[47m                                                                                 \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m**If you’re not having those symptoms:**\u001b[0m\u001b[47m                                                                       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- It could be something less serious (like anxiety, acid reflux, or a muscle strain), but it’s still a good \u001b[0m\u001b[47m   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47midea to check in with a healthcare provider soon—especially since it’s lasted for a couple of days.\u001b[0m\u001b[47m            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Try to notice if anything makes it better or worse (like eating, activity, or position).\u001b[0m\u001b[47m                     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- If you have any risk factors (like high blood pressure, diabetes, smoking, or family history of heart \u001b[0m\u001b[47m       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mproblems), it’s even more important to get checked.\u001b[0m\u001b[47m                                                            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m**Bottom line:**  \u001b[0m\u001b[47m                                                                                             \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mIf you’re worried, it’s always better to be safe and get checked by a doctor. You can call your primary care \u001b[0m\u001b[47m  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mprovider or visit an urgent care clinic if you’re not having severe symptoms. If at any point you feel worse, \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mget emergency help right away.\u001b[0m\u001b[47m                                                                                 \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mWould you like help deciding where to go or what to say to a doctor?\u001b[0m\u001b[47m                                           \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
+       "\u001b[34m╭─\u001b[0m\u001b[34m────────────────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;32mInput\u001b[0m\u001b[34m \u001b[0m\u001b[34m────────────────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;36mI'm bruising a lot on my arms and legs and it's starting to worry me, do you think it could be a blood disorder\u001b[0m \u001b[34m│\u001b[0m\n",
+       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
       ]
      },
      "metadata": {},
@@ -779,69 +789,79 @@
     {
      "data": {
       "text/html": [
-       "
╭───────────────────────────────────────── Output (base reasoning model) ─────────────────────────────────────────╮\n",
-       "│ I’m not a doctor, but chest discomfort—especially if it’s new and lasting for days—really deserves at least a   │\n",
-       "│ phone-call with a medical professional. Here’s a rough guide to help you decide what to do next:                │\n",
-       "│                                                                                                                 │\n",
-       "│ 1. Watch for “red-flag” symptoms. If you develop any of these, call 911 or go straight to the ER:               │\n",
-       "│    • Pain or pressure that spreads to your jaw, arm, shoulder or back                                           │\n",
-       "│    • Crushing or squeezing sensation that doesn’t ease with rest                                                │\n",
-       "│    • Shortness of breath, dizziness, lightheadedness or fainting                                                │\n",
-       "│    • Cold sweat, nausea or feeling “clammy”                                                                     │\n",
-       "│    • Rapid or irregular heartbeat                                                                               │\n",
-       "│                                                                                                                 │\n",
-       "│ 2. If you’re not having those red-flag signs but the pressure is still there after two days:                    │\n",
-       "│    • Call your primary-care doctor or an urgent-care clinic. Explain exactly what you feel, how long it’s       │\n",
-       "│ lasted, and what makes it better or worse.                                                                      │\n",
-       "│    • They may ask you to come in for a quick exam, an EKG or blood tests to rule out heart issues.              │\n",
-       "│                                                                                                                 │\n",
-       "│ 3. Possible non-cardiac causes (which only a clinician can sort out):                                           │\n",
-       "│    • Musculoskeletal strain (costochondritis) – often tender to the touch or when you move your chest/arms      │\n",
-       "│    • Acid reflux or esophageal spasm – sometimes burning, worse after eating or lying down                      │\n",
-       "│    • Anxiety or panic – chest tightness often comes with rapid breathing, palpitations, a sense of dread        │\n",
-       "│                                                                                                                 │\n",
-       "│ 4. In the meantime:                                                                                             │\n",
-       "│    • Keep a symptom diary (when it starts, what you were doing, other sensations you notice)                    │\n",
-       "│    • Try simple stretches or an over-the-counter NSAID (like ibuprofen) if you suspect a pulled muscle          │\n",
-       "│    • Practice slow, deep breathing or relaxation exercises if stress feels like a trigger                       │\n",
-       "│                                                                                                                 │\n",
-       "│ Bottom line: chest pressure isn’t something to tough out on your own. If it’s mild and truly not worsening,     │\n",
-       "│ booking a same-day appointment or going to urgent care is reasonable. But the moment you feel it’s              │\n",
-       "│ intensifying, spreading, or accompanied by any red-flag signs, call emergency services. Better safe than sorry. │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
+       "
╭──────────────────────────────────────────── Output (original model) ────────────────────────────────────────────╮\n",
+       "│ I'm not a doctor, but I can give you some information that might help you decide what to do next.               │\n",
+       "│                                                                                                                 │\n",
+       "│ Bruising easily can have many causes—some are minor, but others can be more serious. Common reasons for         │\n",
+       "│ increased bruising include:                                                                                     │\n",
+       "│                                                                                                                 │\n",
+       "│ - **Aging:** Skin and blood vessels become more fragile.                                                        │\n",
+       "│ - **Medications:** Blood thinners (like aspirin, warfarin), steroids, and some supplements can increase         │\n",
+       "│ bruising.                                                                                                       │\n",
+       "│ - **Nutritional deficiencies:** Low levels of vitamin C, vitamin K, or certain proteins.                        │\n",
+       "│ - **Blood disorders:** Conditions like thrombocytopenia (low platelets), hemophilia, or other clotting          │\n",
+       "│ disorders.                                                                                                      │\n",
+       "│ - **Other medical conditions:** Liver disease, certain infections, or autoimmune diseases.                      │\n",
+       "│                                                                                                                 │\n",
+       "│ **When to be concerned:**                                                                                       │\n",
+       "│ You should see a healthcare provider soon if you notice any of the following:                                   │\n",
+       "│ - Bruises appearing without any known injury.                                                                   │\n",
+       "│ - Large or painful bruises.                                                                                     │\n",
+       "│ - Bruising accompanied by other symptoms (like frequent nosebleeds, bleeding gums, blood in urine/stool,        │\n",
+       "│ fatigue, or unexplained weight loss).                                                                           │\n",
+       "│ - Bruising that seems to be getting worse or spreading.                                                         │\n",
+       "│                                                                                                                 │\n",
+       "│ **What you can do now:**                                                                                        │\n",
+       "│ - Make a note of any new medications or supplements you’ve started.                                             │\n",
+       "│ - Keep track of how many bruises you get and where they appear.                                                 │\n",
+       "│ - Schedule an appointment with your doctor to discuss your symptoms.                                            │\n",
+       "│                                                                                                                 │\n",
+       "│ While it could be something minor, it’s important to get checked out to rule out any serious causes, including  │\n",
+       "│ blood disorders.                                                                                                │\n",
+       "│                                                                                                                 │\n",
+       "│ If you develop severe symptoms, such as difficulty breathing, severe headache, or uncontrolled bleeding, seek   │\n",
+       "│ emergency care immediately.                                                                                     │\n",
+       "│                                                                                                                 │\n",
+       "│ Would you like more information about what to expect at your doctor's visit or how to prepare?                  │\n",
+       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
        "
\n"
       ],
       "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m────────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mOutput (base reasoning model)\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m────────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mI’m not a doctor, but chest discomfort—especially if it’s new and lasting for days—really deserves at least a \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mphone-call with a medical professional. Here’s a rough guide to help you decide what to do next:\u001b[0m\u001b[47m               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m1. Watch for “red-flag” symptoms. If you develop any of these, call 911 or go straight to the ER:  \u001b[0m\u001b[47m            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Pain or pressure that spreads to your jaw, arm, shoulder or back  \u001b[0m\u001b[47m                                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Crushing or squeezing sensation that doesn’t ease with rest  \u001b[0m\u001b[47m                                             \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Shortness of breath, dizziness, lightheadedness or fainting  \u001b[0m\u001b[47m                                             \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Cold sweat, nausea or feeling “clammy”  \u001b[0m\u001b[47m                                                                  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Rapid or irregular heartbeat  \u001b[0m\u001b[47m                                                                            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m2. If you’re not having those red-flag signs but the pressure is still there after two days:  \u001b[0m\u001b[47m                 \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Call your primary-care doctor or an urgent-care clinic. Explain exactly what you feel, how long it’s \u001b[0m\u001b[47m     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mlasted, and what makes it better or worse.  \u001b[0m\u001b[47m                                                                   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • They may ask you to come in for a quick exam, an EKG or blood tests to rule out heart issues.  \u001b[0m\u001b[47m           \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m3. Possible non-cardiac causes (which only a clinician can sort out):  \u001b[0m\u001b[47m                                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Musculoskeletal strain (costochondritis) – often tender to the touch or when you move your chest/arms  \u001b[0m\u001b[47m   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Acid reflux or esophageal spasm – sometimes burning, worse after eating or lying down  \u001b[0m\u001b[47m                   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Anxiety or panic – chest tightness often comes with rapid breathing, palpitations, a sense of dread  \u001b[0m\u001b[47m     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m4. In the meantime:  \u001b[0m\u001b[47m                                                                                          \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Keep a symptom diary (when it starts, what you were doing, other sensations you notice)  \u001b[0m\u001b[47m                 \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Try simple stretches or an over-the-counter NSAID (like ibuprofen) if you suspect a pulled muscle  \u001b[0m\u001b[47m       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m   • Practice slow, deep breathing or relaxation exercises if stress feels like a trigger  \u001b[0m\u001b[47m                    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mBottom line: chest pressure isn’t something to tough out on your own. If it’s mild and truly not worsening, \u001b[0m\u001b[47m   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mbooking a same-day appointment or going to urgent care is reasonable. But the moment you feel it’s \u001b[0m\u001b[47m            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mintensifying, spreading, or accompanied by any red-flag signs, call emergency services. Better safe than sorry.\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
+       "\u001b[34m╭─\u001b[0m\u001b[34m───────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;32mOutput (original model)\u001b[0m\u001b[34m \u001b[0m\u001b[34m───────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mI'm not a doctor, but I can give you some information that might help you decide what to do next.\u001b[0m               \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mBruising easily can have many causes—some are minor, but others can be more serious. Common reasons for \u001b[0m        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mincreased bruising include:\u001b[0m                                                                                     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **Aging:** Skin and blood vessels become more fragile.\u001b[0m                                                        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **Medications:** Blood thinners (like aspirin, warfarin), steroids, and some supplements can increase \u001b[0m        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mbruising.\u001b[0m                                                                                                       \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **Nutritional deficiencies:** Low levels of vitamin C, vitamin K, or certain proteins.\u001b[0m                        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **Blood disorders:** Conditions like thrombocytopenia (low platelets), hemophilia, or other clotting \u001b[0m         \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mdisorders.\u001b[0m                                                                                                      \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **Other medical conditions:** Liver disease, certain infections, or autoimmune diseases.\u001b[0m                      \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**When to be concerned:**  \u001b[0m                                                                                     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mYou should see a healthcare provider soon if you notice any of the following:\u001b[0m                                   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Bruises appearing without any known injury.\u001b[0m                                                                   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Large or painful bruises.\u001b[0m                                                                                     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Bruising accompanied by other symptoms (like frequent nosebleeds, bleeding gums, blood in urine/stool, \u001b[0m       \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mfatigue, or unexplained weight loss).\u001b[0m                                                                           \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Bruising that seems to be getting worse or spreading.\u001b[0m                                                         \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**What you can do now:**\u001b[0m                                                                                        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Make a note of any new medications or supplements you’ve started.\u001b[0m                                             \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Keep track of how many bruises you get and where they appear.\u001b[0m                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Schedule an appointment with your doctor to discuss your symptoms.\u001b[0m                                            \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mWhile it could be something minor, it’s important to get checked out to rule out any serious causes, including \u001b[0m \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mblood disorders.\u001b[0m                                                                                                \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mIf you develop severe symptoms, such as difficulty breathing, severe headache, or uncontrolled bleeding, seek \u001b[0m  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33memergency care immediately.\u001b[0m                                                                                     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mWould you like more information about what to expect at your doctor's visit or how to prepare?\u001b[0m                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
       ]
      },
      "metadata": {},
@@ -850,47 +870,43 @@
     {
      "data": {
       "text/html": [
-       "
╭────────────────────────────────────── Output (fine-tuned reasoning model) ──────────────────────────────────────╮\n",
-       "│ I’m sorry you’re going through that—let’s see if we can figure out what’s going on. A few quick questions:      │\n",
-       "│                                                                                                                 │\n",
-       "│ 1. How old are you, and do you have any medical conditions (high blood pressure, high cholesterol, diabetes,    │\n",
-       "│ asthma, anxiety, etc.)?                                                                                         │\n",
-       "│ 2. Exactly where do you feel the pressure—center of your chest, left side, under your breastbone?               │\n",
-       "│ 3. Can you describe it—dull ache, sharp stabbing, squeezing, burning?                                           │\n",
-       "│ 4. Does it ever radiate—to your arm, jaw, back?                                                                 │\n",
-       "│ 5. What were you doing when it started, and does any activity (walking, climbing stairs, lifting) make it worse │\n",
-       "│ or better?                                                                                                      │\n",
-       "│ 6. Does it change if you take a deep breath, cough, press on your chest, or change position?                    │\n",
-       "│ 7. Any other symptoms—palpitations (heart racing/skipping), dizziness, sweating, nausea, cough, heartburn, or   │\n",
-       "│ leg swelling?                                                                                                   │\n",
-       "│ 8. Have you noticed any triggers or stressors recently?                                                         │\n",
-       "│                                                                                                                 │\n",
-       "│ If at any point you feel it’s crushing, comes on suddenly, or you develop sweating, nausea, shortness of        │\n",
-       "│ breath, lightheadedness, or pain radiating to your arm/jaw—even briefly—call 911 or go to the nearest ER right  │\n",
-       "│ away. Let me know the answers and we’ll go from there.                                                          │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
+       "
╭─────────────────────────────────────────── Output (fine-tuned model) ───────────────────────────────────────────╮\n",
+       "│ I’m sorry you’re experiencing this—it can certainly be concerning. To help figure out whether your bruising     │\n",
+       "│ might be from a blood‑related issue or something else, could you tell me a bit more about what you’re seeing    │\n",
+       "│ and your health history? For example:                                                                           │\n",
+       "│ 1. When did you first notice these bruises, and how often are new ones appearing?                               │\n",
+       "│ 2. Do they occur after any bump or injury, or do they seem to arise spontaneously?                              │\n",
+       "│ 3. What do the bruises look like in terms of size, color, and number?                                           │\n",
+       "│ 4. Have you started or stopped any medications or supplements recently (especially blood thinners, NSAIDs,      │\n",
+       "│ aspirin, steroids, or herbal supplements)?                                                                      │\n",
+       "│ 5. Do you have any other bleeding symptoms—nosebleeds, bleeding gums, unusually heavy menstrual periods, blood  │\n",
+       "│ in stool or urine?                                                                                              │\n",
+       "│ 6. Have you experienced other symptoms such as fatigue, fever, weight loss, or night sweats?                    │\n",
+       "│ 7. Is there any family history of bruising easily, bleeding disorders, or clotting abnormalities?               │\n",
+       "│                                                                                                                 │\n",
+       "│ With that information, I can better suggest whether it’s appropriate to check things like your platelet count,  │\n",
+       "│ clotting factors, or other tests, or whether it might be related to something less worrisome.                   │\n",
+       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
        "
\n"
       ],
       "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m─────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mOutput (fine-tuned reasoning model)\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m─────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mI’m sorry you’re going through that—let’s see if we can figure out what’s going on. A few quick questions:\u001b[0m\u001b[47m     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m1. How old are you, and do you have any medical conditions (high blood pressure, high cholesterol, diabetes, \u001b[0m\u001b[47m  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47masthma, anxiety, etc.)?\u001b[0m\u001b[47m                                                                                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m2. Exactly where do you feel the pressure—center of your chest, left side, under your breastbone?\u001b[0m\u001b[47m              \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m3. Can you describe it—dull ache, sharp stabbing, squeezing, burning?\u001b[0m\u001b[47m                                          \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m4. Does it ever radiate—to your arm, jaw, back?\u001b[0m\u001b[47m                                                                \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m5. What were you doing when it started, and does any activity (walking, climbing stairs, lifting) make it worse\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mor better?\u001b[0m\u001b[47m                                                                                                     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m6. Does it change if you take a deep breath, cough, press on your chest, or change position?\u001b[0m\u001b[47m                   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m7. Any other symptoms—palpitations (heart racing/skipping), dizziness, sweating, nausea, cough, heartburn, or \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mleg swelling?\u001b[0m\u001b[47m                                                                                                  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m8. Have you noticed any triggers or stressors recently?\u001b[0m\u001b[47m                                                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mIf at any point you feel it’s crushing, comes on suddenly, or you develop sweating, nausea, shortness of \u001b[0m\u001b[47m      \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mbreath, lightheadedness, or pain radiating to your arm/jaw—even briefly—call 911 or go to the nearest ER right \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47maway. Let me know the answers and we’ll go from there.\u001b[0m\u001b[47m                                                         \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
+       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;32mOutput (fine-tuned model)\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mI’m sorry you’re experiencing this—it can certainly be concerning. To help figure out whether your bruising \u001b[0m    \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mmight be from a blood‑related issue or something else, could you tell me a bit more about what you’re seeing \u001b[0m   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mand your health history? For example:  \u001b[0m                                                                         \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m1. When did you first notice these bruises, and how often are new ones appearing?  \u001b[0m                             \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m2. Do they occur after any bump or injury, or do they seem to arise spontaneously?  \u001b[0m                            \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m3. What do the bruises look like in terms of size, color, and number?  \u001b[0m                                         \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m4. Have you started or stopped any medications or supplements recently (especially blood thinners, NSAIDs, \u001b[0m     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35maspirin, steroids, or herbal supplements)?  \u001b[0m                                                                    \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m5. Do you have any other bleeding symptoms—nosebleeds, bleeding gums, unusually heavy menstrual periods, blood \u001b[0m \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35min stool or urine?  \u001b[0m                                                                                            \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m6. Have you experienced other symptoms such as fatigue, fever, weight loss, or night sweats?  \u001b[0m                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m7. Is there any family history of bruising easily, bleeding disorders, or clotting abnormalities?  \u001b[0m             \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mWith that information, I can better suggest whether it’s appropriate to check things like your platelet count, \u001b[0m \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mclotting factors, or other tests, or whether it might be related to something less worrisome.\u001b[0m                   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
       ]
      },
      "metadata": {},
@@ -916,96 +932,15 @@
     {
      "data": {
       "text/html": [
-       "
╭───────────────────────────────────────────────────── Input ─────────────────────────────────────────────────────╮\n",
-       "│ I'm bruising a lot on my arms and legs and it's starting to worry me, do you think it could be a blood disorder │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m────────────────────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mInput\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m────────────────────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[30;47mI'm bruising a lot on my arms and legs and it's starting to worry me, do you think it could be a blood disorder\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
╭────────────────────────────────────── Output (original model completion) ───────────────────────────────────────╮\n",
-       "│ I'm not a doctor, but I can give you some information that might help you decide what to do next.               │\n",
-       "│                                                                                                                 │\n",
-       "│ Bruising easily can have many causes—some are minor, but others can be more serious. Common reasons for         │\n",
-       "│ increased bruising include:                                                                                     │\n",
-       "│                                                                                                                 │\n",
-       "│ - **Aging:** Skin and blood vessels become more fragile.                                                        │\n",
-       "│ - **Medications:** Blood thinners (like aspirin, warfarin), steroids, and some supplements can increase         │\n",
-       "│ bruising.                                                                                                       │\n",
-       "│ - **Nutritional deficiencies:** Low levels of vitamin C, vitamin K, or certain proteins.                        │\n",
-       "│ - **Blood disorders:** Conditions like thrombocytopenia (low platelets), hemophilia, or other clotting          │\n",
-       "│ disorders.                                                                                                      │\n",
-       "│ - **Other medical conditions:** Liver disease, certain infections, or autoimmune diseases.                      │\n",
-       "│                                                                                                                 │\n",
-       "│ **When to be concerned:**                                                                                       │\n",
-       "│ You should see a healthcare provider soon if you notice any of the following:                                   │\n",
-       "│ - Bruises appearing without any known injury.                                                                   │\n",
-       "│ - Large or painful bruises.                                                                                     │\n",
-       "│ - Bruising accompanied by other symptoms (like frequent nosebleeds, bleeding gums, blood in urine/stool,        │\n",
-       "│ fatigue, or unexplained weight loss).                                                                           │\n",
-       "│ - Bruising that seems to be getting worse or spreading.                                                         │\n",
-       "│                                                                                                                 │\n",
-       "│ **What you can do now:**                                                                                        │\n",
-       "│ - Make a note of any new medications or supplements you’ve started.                                             │\n",
-       "│ - Keep track of how many bruises you get and where they appear.                                                 │\n",
-       "│ - Schedule an appointment with your doctor to discuss your symptoms.                                            │\n",
-       "│                                                                                                                 │\n",
-       "│ While it could be something minor, it’s important to get checked out to rule out any serious causes, including  │\n",
-       "│ blood disorders.                                                                                                │\n",
-       "│                                                                                                                 │\n",
-       "│ If you develop severe symptoms, such as difficulty breathing, severe headache, or uncontrolled bleeding, seek   │\n",
-       "│ emergency care immediately.                                                                                     │\n",
-       "│                                                                                                                 │\n",
-       "│ Would you like more information about what to expect at your doctor's visit or how to prepare?                  │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
+       "
╭───────────────────────────────────────────────────── Input ─────────────────────────────────────────────────────╮\n",
+       "│ adult routine cholesterol screening guidelines                                                                  │\n",
+       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
        "
\n"
       ],
       "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m─────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mOutput (original model completion)\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m──────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mI'm not a doctor, but I can give you some information that might help you decide what to do next.\u001b[0m\u001b[47m              \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mBruising easily can have many causes—some are minor, but others can be more serious. Common reasons for \u001b[0m\u001b[47m       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mincreased bruising include:\u001b[0m\u001b[47m                                                                                    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- **Aging:** Skin and blood vessels become more fragile.\u001b[0m\u001b[47m                                                       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- **Medications:** Blood thinners (like aspirin, warfarin), steroids, and some supplements can increase \u001b[0m\u001b[47m       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mbruising.\u001b[0m\u001b[47m                                                                                                      \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- **Nutritional deficiencies:** Low levels of vitamin C, vitamin K, or certain proteins.\u001b[0m\u001b[47m                       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- **Blood disorders:** Conditions like thrombocytopenia (low platelets), hemophilia, or other clotting \u001b[0m\u001b[47m        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mdisorders.\u001b[0m\u001b[47m                                                                                                     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- **Other medical conditions:** Liver disease, certain infections, or autoimmune diseases.\u001b[0m\u001b[47m                     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m**When to be concerned:**  \u001b[0m\u001b[47m                                                                                    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mYou should see a healthcare provider soon if you notice any of the following:\u001b[0m\u001b[47m                                  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Bruises appearing without any known injury.\u001b[0m\u001b[47m                                                                  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Large or painful bruises.\u001b[0m\u001b[47m                                                                                    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Bruising accompanied by other symptoms (like frequent nosebleeds, bleeding gums, blood in urine/stool, \u001b[0m\u001b[47m      \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mfatigue, or unexplained weight loss).\u001b[0m\u001b[47m                                                                          \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Bruising that seems to be getting worse or spreading.\u001b[0m\u001b[47m                                                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m**What you can do now:**\u001b[0m\u001b[47m                                                                                       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Make a note of any new medications or supplements you’ve started.\u001b[0m\u001b[47m                                            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Keep track of how many bruises you get and where they appear.\u001b[0m\u001b[47m                                                \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47m- Schedule an appointment with your doctor to discuss your symptoms.\u001b[0m\u001b[47m                                           \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mWhile it could be something minor, it’s important to get checked out to rule out any serious causes, including \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mblood disorders.\u001b[0m\u001b[47m                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mIf you develop severe symptoms, such as difficulty breathing, severe headache, or uncontrolled bleeding, seek \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47memergency care immediately.\u001b[0m\u001b[47m                                                                                    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[34;47mWould you like more information about what to expect at your doctor's visit or how to prepare?\u001b[0m\u001b[47m                 \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
+       "\u001b[34m╭─\u001b[0m\u001b[34m────────────────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;32mInput\u001b[0m\u001b[34m \u001b[0m\u001b[34m────────────────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;36madult routine cholesterol screening guidelines\u001b[0m                                                                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
       ]
      },
      "metadata": {},
@@ -1014,123 +949,129 @@
     {
      "data": {
       "text/html": [
-       "
╭───────────────────────────────────────── Output (base reasoning model) ─────────────────────────────────────────╮\n",
-       "│ I’m not a doctor, but I can share some information that may help you decide whether to seek medical attention   │\n",
-       "│ and what questions to ask your provider.                                                                        │\n",
-       "│                                                                                                                 │\n",
-       "│ 1. Common causes of easy bruising                                                                               │\n",
-       "│   • Minor bumps or knocks you don’t remember                                                                    │\n",
-       "│   • Medications or supplements that thin your blood (aspirin, NSAIDs, fish oil, ginko, warfarin, direct oral    │\n",
-       "│ anticoagulants)                                                                                                 │\n",
-       "│   • Aging skin and blood vessels (the skin thins and capillaries become more fragile)                           │\n",
-       "│                                                                                                                 │\n",
-       "│ 2. Possible medical causes to discuss with your doctor                                                          │\n",
-       "│   a) Platelet problems                                                                                          │\n",
-       "│   – Low platelet count (thrombocytopenia) from immune causes (e.g. ITP), infections, certain medications        │\n",
-       "│   – Platelet dysfunction (e.g. von Willebrand disease, inherited or acquired disorders)                         │\n",
-       "│   b) Clotting-factor deficiencies                                                                               │\n",
-       "│   – Hemophilia A or B (rare in women)                                                                           │\n",
-       "│   – Vitamin K deficiency (malabsorption, certain antibiotics)                                                   │\n",
-       "│   c) Liver disease                                                                                              │\n",
-       "│   – The liver makes many clotting factors; if it’s not working well you can bruise more easily                  │\n",
-       "│   d) Vascular fragility                                                                                         │\n",
-       "│   – Vasculitis or connective-tissue disorders that weaken blood vessel walls                                    │\n",
-       "│   e) Nutritional deficiencies                                                                                   │\n",
-       "│   – Vitamin C (scurvy), vitamin K, or protein deficiency                                                        │\n",
-       "│                                                                                                                 │\n",
-       "│ 3. Red-flag symptoms that warrant prompt evaluation                                                             │\n",
-       "│   • Bruises appearing without any remembered bump or injury                                                     │\n",
-       "│   • Large “hematomas” (deep, painful swellings under the skin)                                                  │\n",
-       "│   • Small red/purple flat spots (petechiae) or pinpoint bleeds around the hair follicles                        │\n",
-       "│   • Bleeding gums, frequent nosebleeds, blood in stool or urine, unusually heavy menstrual bleeding             │\n",
-       "│   • Unexplained weight loss, night sweats, fevers (could point toward an underlying illness)                    │\n",
-       "│                                                                                                                 │\n",
-       "│ 4. What you can do now                                                                                          │\n",
-       "│   • Keep a “bruise diary”: note when and where each bruise appears, how big it is, and any associated symptoms. │\n",
-       "│   • Review any medications or supplements you take—ask your pharmacist or provider if they affect bleeding      │\n",
-       "│ risk.                                                                                                           │\n",
-       "│   • Make sure your diet includes adequate protein, vitamin C (citrus fruits, berries, peppers), and vitamin K   │\n",
-       "│ (leafy greens).                                                                                                 │\n",
-       "│                                                                                                                 │\n",
-       "│ 5. When to see a doctor                                                                                         │\n",
-       "│   • Bruising is significantly more frequent or severe than you can explain by bumps and knocks                  │\n",
-       "│   • You have any of the red-flag symptoms above                                                                 │\n",
-       "│   • You’re on a blood thinner and your bruising seems out of proportion                                         │\n",
-       "│                                                                                                                 │\n",
-       "│ 6. What your doctor may do                                                                                      │\n",
-       "│   • Physical exam (skin inspection, signs of liver disease, spleen enlargement)                                 │\n",
-       "│   • Blood tests:                                                                                                │\n",
-       "│     – Complete blood count (CBC) with platelet count                                                            │\n",
-       "│     – Coagulation panel (PT/INR, aPTT)                                                                          │\n",
-       "│     – Liver-function tests                                                                                      │\n",
-       "│     – Specific factor levels or von Willebrand assay if indicated                                               │\n",
-       "│   • Referral to a hematologist if an inherited or serious acquired disorder is suspected                        │\n",
-       "│                                                                                                                 │\n",
-       "│ Bottom line: occasional bruising is common, especially if you bump into things or take mild blood thinners. But │\n",
-       "│ if your bruising is frequent, spontaneous, or accompanied by other bleeding symptoms, you should get evaluated. │\n",
-       "│ A primary-care doctor can order simple blood tests to rule out the most common disorders and guide you to any   │\n",
-       "│ needed specialist care.                                                                                         │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
+       "
╭──────────────────────────────────────────── Output (original model) ────────────────────────────────────────────╮\n",
+       "│ Here is a summary of current guidelines for adult routine cholesterol screening:                                │\n",
+       "│                                                                                                                 │\n",
+       "│ **General Recommendations:**                                                                                    │\n",
+       "│                                                                                                                 │\n",
+       "│ - **All adults aged 20 years or older**: The American Heart Association (AHA), American College of Cardiology   │\n",
+       "│ (ACC), and U.S. Preventive Services Task Force (USPSTF) recommend routine cholesterol screening starting at age │\n",
+       "│ 20, with repeat testing every 4–6 years if risk remains low.                                                    │\n",
+       "│                                                                                                                 │\n",
+       "│ **More Specific Guidelines:**                                                                                   │\n",
+       "│                                                                                                                 │\n",
+       "│ ### U.S. Preventive Services Task Force (USPSTF) (2016):                                                        │\n",
+       "│                                                                                                                 │\n",
+       "│ - **Adults aged 40–75**: Strongly recommend screening.                                                          │\n",
+       "│ - **Adults aged 20–39**: Consider screening if they have risk factors for cardiovascular disease (e.g.,         │\n",
+       "│ diabetes, hypertension, family history of early heart disease, smoking, obesity).                               │\n",
+       "│ - **Frequency**: Every 4–6 years for low-risk individuals; more frequently if risk factors are present.         │\n",
+       "│                                                                                                                 │\n",
+       "│ ### American College of Cardiology (ACC)/American Heart Association (AHA) (2018):                               │\n",
+       "│                                                                                                                 │\n",
+       "│ - **Adults aged 20 and older**: Assess cholesterol as part of cardiovascular risk assessment every 4–6 years.   │\n",
+       "│ - **More frequent testing**: For those with risk factors (e.g., diabetes, hypertension, family history,         │\n",
+       "│ obesity) or those on cholesterol-lowering therapy.                                                              │\n",
+       "│                                                                                                                 │\n",
+       "│ ### National Lipid Association (NLA):                                                                           │\n",
+       "│                                                                                                                 │\n",
+       "│ - **All adults 20 years and older**: Lipid profile at least every 5 years.                                      │\n",
+       "│ - **Earlier and/or more frequent testing**: If risk factors or family history of premature atherosclerotic      │\n",
+       "│ cardiovascular disease (ASCVD).                                                                                 │\n",
+       "│                                                                                                                 │\n",
+       "│ **What is measured?**                                                                                           │\n",
+       "│ - A standard fasting or non-fasting lipid panel measures:                                                       │\n",
+       "│   - Total cholesterol                                                                                           │\n",
+       "│   - LDL cholesterol (\"bad\")                                                                                     │\n",
+       "│   - HDL cholesterol (\"good\")                                                                                    │\n",
+       "│   - Triglycerides                                                                                               │\n",
+       "│                                                                                                                 │\n",
+       "│ **Summary Table:**                                                                                              │\n",
+       "│                                                                                                                 │\n",
+       "│ | Age Group         | Routine Screening? | Frequency      | More Frequent If...                |                │\n",
+       "│ |-------------------|-------------------|---------------|------------------------------------|                  │\n",
+       "│ | 20–39 years       | Consider if risk  | 4–6 years     | Risk factors present               |                  │\n",
+       "│ | 40–75 years       | Yes               | 4–6 years     | Risk factors or on therapy         |                  │\n",
+       "│ | >75 years         | Individualized    | Case-by-case  | Based on overall health/risk       |                  │\n",
+       "│                                                                                                                 │\n",
+       "│ **Key Risk Factors:**                                                                                           │\n",
+       "│ - Diabetes                                                                                                      │\n",
+       "│ - Hypertension                                                                                                  │\n",
+       "│ - Smoking                                                                                                       │\n",
+       "│ - Family history of early heart disease                                                                         │\n",
+       "│ - Obesity                                                                                                       │\n",
+       "│                                                                                                                 │\n",
+       "│ **References:**                                                                                                 │\n",
+       "│ - 2018 ACC/AHA Guideline on the Management of Blood Cholesterol                                                 │\n",
+       "│ - USPSTF Recommendation Statement (2016)                                                                        │\n",
+       "│                                                                                                                 │\n",
+       "│ **Note:** These are general recommendations. Screening intervals and starting age may be adjusted based on      │\n",
+       "│ individual risk factors and clinical judgment. Always consult with a healthcare provider for personalized       │\n",
+       "│ advice.                                                                                                         │\n",
+       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
        "
\n"
       ],
       "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m────────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mOutput (base reasoning model)\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m────────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mI’m not a doctor, but I can share some information that may help you decide whether to seek medical attention \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mand what questions to ask your provider.\u001b[0m\u001b[47m                                                                       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m1. Common causes of easy bruising  \u001b[0m\u001b[47m                                                                            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Minor bumps or knocks you don’t remember  \u001b[0m\u001b[47m                                                                 \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Medications or supplements that thin your blood (aspirin, NSAIDs, fish oil, ginko, warfarin, direct oral \u001b[0m\u001b[47m  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47manticoagulants)  \u001b[0m\u001b[47m                                                                                              \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Aging skin and blood vessels (the skin thins and capillaries become more fragile)  \u001b[0m\u001b[47m                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m2. Possible medical causes to discuss with your doctor  \u001b[0m\u001b[47m                                                       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  a) Platelet problems  \u001b[0m\u001b[47m                                                                                       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  – Low platelet count (thrombocytopenia) from immune causes (e.g. ITP), infections, certain medications  \u001b[0m\u001b[47m     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  – Platelet dysfunction (e.g. von Willebrand disease, inherited or acquired disorders)  \u001b[0m\u001b[47m                      \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  b) Clotting-factor deficiencies  \u001b[0m\u001b[47m                                                                            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  – Hemophilia A or B (rare in women)  \u001b[0m\u001b[47m                                                                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  – Vitamin K deficiency (malabsorption, certain antibiotics)  \u001b[0m\u001b[47m                                                \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  c) Liver disease  \u001b[0m\u001b[47m                                                                                           \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  – The liver makes many clotting factors; if it’s not working well you can bruise more easily  \u001b[0m\u001b[47m               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  d) Vascular fragility  \u001b[0m\u001b[47m                                                                                      \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  – Vasculitis or connective-tissue disorders that weaken blood vessel walls  \u001b[0m\u001b[47m                                 \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  e) Nutritional deficiencies  \u001b[0m\u001b[47m                                                                                \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  – Vitamin C (scurvy), vitamin K, or protein deficiency  \u001b[0m\u001b[47m                                                     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m3. Red-flag symptoms that warrant prompt evaluation  \u001b[0m\u001b[47m                                                          \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Bruises appearing without any remembered bump or injury  \u001b[0m\u001b[47m                                                  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Large “hematomas” (deep, painful swellings under the skin)  \u001b[0m\u001b[47m                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Small red/purple flat spots (petechiae) or pinpoint bleeds around the hair follicles  \u001b[0m\u001b[47m                     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Bleeding gums, frequent nosebleeds, blood in stool or urine, unusually heavy menstrual bleeding  \u001b[0m\u001b[47m          \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Unexplained weight loss, night sweats, fevers (could point toward an underlying illness)  \u001b[0m\u001b[47m                 \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m4. What you can do now  \u001b[0m\u001b[47m                                                                                       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Keep a “bruise diary”: note when and where each bruise appears, how big it is, and any associated symptoms.\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Review any medications or supplements you take—ask your pharmacist or provider if they affect bleeding \u001b[0m\u001b[47m    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mrisk.  \u001b[0m\u001b[47m                                                                                                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Make sure your diet includes adequate protein, vitamin C (citrus fruits, berries, peppers), and vitamin K \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m(leafy greens).  \u001b[0m\u001b[47m                                                                                              \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m5. When to see a doctor  \u001b[0m\u001b[47m                                                                                      \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Bruising is significantly more frequent or severe than you can explain by bumps and knocks  \u001b[0m\u001b[47m               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • You have any of the red-flag symptoms above  \u001b[0m\u001b[47m                                                              \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • You’re on a blood thinner and your bruising seems out of proportion  \u001b[0m\u001b[47m                                      \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m6. What your doctor may do  \u001b[0m\u001b[47m                                                                                   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Physical exam (skin inspection, signs of liver disease, spleen enlargement)  \u001b[0m\u001b[47m                              \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Blood tests:  \u001b[0m\u001b[47m                                                                                             \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m    – Complete blood count (CBC) with platelet count  \u001b[0m\u001b[47m                                                         \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m    – Coagulation panel (PT/INR, aPTT)  \u001b[0m\u001b[47m                                                                       \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m    – Liver-function tests  \u001b[0m\u001b[47m                                                                                   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m    – Specific factor levels or von Willebrand assay if indicated  \u001b[0m\u001b[47m                                            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47m  • Referral to a hematologist if an inherited or serious acquired disorder is suspected  \u001b[0m\u001b[47m                     \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mBottom line: occasional bruising is common, especially if you bump into things or take mild blood thinners. But\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mif your bruising is frequent, spontaneous, or accompanied by other bleeding symptoms, you should get evaluated.\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mA primary-care doctor can order simple blood tests to rule out the most common disorders and guide you to any \u001b[0m\u001b[47m \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[38;5;22;47mneeded specialist care.\u001b[0m\u001b[47m                                                                                        \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
+       "\u001b[34m╭─\u001b[0m\u001b[34m───────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;32mOutput (original model)\u001b[0m\u001b[34m \u001b[0m\u001b[34m───────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mHere is a summary of current guidelines for adult routine cholesterol screening:\u001b[0m                                \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**General Recommendations:**\u001b[0m                                                                                    \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **All adults aged 20 years or older**: The American Heart Association (AHA), American College of Cardiology \u001b[0m  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m(ACC), and U.S. Preventive Services Task Force (USPSTF) recommend routine cholesterol screening starting at age\u001b[0m \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m20, with repeat testing every 4–6 years if risk remains low.\u001b[0m                                                    \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**More Specific Guidelines:**\u001b[0m                                                                                   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m### U.S. Preventive Services Task Force (USPSTF) (2016):\u001b[0m                                                        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **Adults aged 40–75**: Strongly recommend screening.\u001b[0m                                                          \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **Adults aged 20–39**: Consider screening if they have risk factors for cardiovascular disease (e.g., \u001b[0m        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mdiabetes, hypertension, family history of early heart disease, smoking, obesity).\u001b[0m                               \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **Frequency**: Every 4–6 years for low-risk individuals; more frequently if risk factors are present.\u001b[0m         \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m### American College of Cardiology (ACC)/American Heart Association (AHA) (2018):\u001b[0m                               \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **Adults aged 20 and older**: Assess cholesterol as part of cardiovascular risk assessment every 4–6 years.\u001b[0m   \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **More frequent testing**: For those with risk factors (e.g., diabetes, hypertension, family history, \u001b[0m        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mobesity) or those on cholesterol-lowering therapy.\u001b[0m                                                              \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m### National Lipid Association (NLA):\u001b[0m                                                                           \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **All adults 20 years and older**: Lipid profile at least every 5 years.\u001b[0m                                      \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- **Earlier and/or more frequent testing**: If risk factors or family history of premature atherosclerotic \u001b[0m     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mcardiovascular disease (ASCVD).\u001b[0m                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**What is measured?**\u001b[0m                                                                                           \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- A standard fasting or non-fasting lipid panel measures:\u001b[0m                                                       \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m  - Total cholesterol\u001b[0m                                                                                           \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m  - LDL cholesterol (\"bad\")\u001b[0m                                                                                     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m  - HDL cholesterol (\"good\")\u001b[0m                                                                                    \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m  - Triglycerides\u001b[0m                                                                                               \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**Summary Table:**\u001b[0m                                                                                              \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m| Age Group         | Routine Screening? | Frequency      | More Frequent If...                |\u001b[0m                \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m|-------------------|-------------------|---------------|------------------------------------|\u001b[0m                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m| 20–39 years       | Consider if risk  | 4–6 years     | Risk factors present               |\u001b[0m                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m| 40–75 years       | Yes               | 4–6 years     | Risk factors or on therapy         |\u001b[0m                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m| >75 years         | Individualized    | Case-by-case  | Based on overall health/risk       |\u001b[0m                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**Key Risk Factors:**\u001b[0m                                                                                           \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Diabetes\u001b[0m                                                                                                      \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Hypertension\u001b[0m                                                                                                  \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Smoking\u001b[0m                                                                                                       \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Family history of early heart disease\u001b[0m                                                                         \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- Obesity\u001b[0m                                                                                                       \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**References:**\u001b[0m                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- 2018 ACC/AHA Guideline on the Management of Blood Cholesterol\u001b[0m                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m- USPSTF Recommendation Statement (2016)\u001b[0m                                                                        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33m**Note:** These are general recommendations. Screening intervals and starting age may be adjusted based on \u001b[0m     \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33mindividual risk factors and clinical judgment. Always consult with a healthcare provider for personalized \u001b[0m      \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;33madvice.\u001b[0m                                                                                                         \u001b[34m│\u001b[0m\n",
+       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
       ]
      },
      "metadata": {},
@@ -1139,77 +1080,29 @@
     {
      "data": {
       "text/html": [
-       "
╭────────────────────────────────────── Output (fine-tuned reasoning model) ──────────────────────────────────────╮\n",
-       "│ Let’s start by getting a clearer picture of your bruising and overall health. I’ll ask you a few questions—your │\n",
-       "│ answers will help me decide whether we need to check for a bleeding disorder or consider other causes:          │\n",
-       "│                                                                                                                 │\n",
-       "│ 1. Onset & Frequency                                                                                            │\n",
-       "│  • When did you first notice the bruising?                                                                      │\n",
-       "│  • How many new bruises do you get in a week?                                                                   │\n",
-       "│                                                                                                                 │\n",
-       "│ 2. Trigger & Distribution                                                                                       │\n",
-       "│  • Do these bruises come up after bumps or minor knocks, or do they appear seemingly out of nowhere?            │\n",
-       "│  • Are they mostly on pressure points (shins, forearms) or also on areas you wouldn’t expect to hit?            │\n",
-       "│                                                                                                                 │\n",
-       "│ 3. Bruise Characteristics                                                                                       │\n",
-       "│  • How large are they, on average?                                                                              │\n",
-       "│  • Do they change color normally (purple → green → yellow) and heal in a few weeks?                             │\n",
-       "│                                                                                                                 │\n",
-       "│ 4. Other Bleeding Symptoms                                                                                      │\n",
-       "│  • Any easy bleeding of gums, frequent nosebleeds, or heavy periods (if applicable)?                            │\n",
-       "│  • Any blood in your urine or stool?                                                                            │\n",
-       "│                                                                                                                 │\n",
-       "│ 5. Medications & Supplements                                                                                    │\n",
-       "│  • Are you taking any blood thinners (warfarin, heparin, DOACs), aspirin, NSAIDs, or herbal supplements (e.g.,  │\n",
-       "│ fish oil, ginkgo)?                                                                                              │\n",
-       "│                                                                                                                 │\n",
-       "│ 6. Medical & Family History                                                                                     │\n",
-       "│  • Do you have liver disease, kidney disease, or a history of malignancy?                                       │\n",
-       "│  • Any family history of easy bruising or known bleeding disorders?                                             │\n",
-       "│                                                                                                                 │\n",
-       "│ 7. Systemic Symptoms                                                                                            │\n",
-       "│  • Have you noticed fatigue, weight loss, fevers, or night sweats?                                              │\n",
-       "│                                                                                                                 │\n",
-       "│ Once I have this information, we can decide whether to check a platelet count, coagulation studies (PT/INR,     │\n",
-       "│ aPTT), and possibly von Willebrand factor levels. Let me know what you’ve observed.                             │\n",
-       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
+       "
╭─────────────────────────────────────────── Output (fine-tuned model) ───────────────────────────────────────────╮\n",
+       "│ Could you help me narrow this down so I can give you the most relevant recommendation? Specifically:            │\n",
+       "│                                                                                                                 │\n",
+       "│ 1. Which guideline or region are you interested in (for example, USPSTF in the US, ACC/AHA, Canadian, European, │\n",
+       "│ etc.)?                                                                                                          │\n",
+       "│ 2. Are we talking about primary‐prevention screening in an asymptomatic adult, or secondary‑prevention          │\n",
+       "│ monitoring in someone with known cardiovascular disease?                                                        │\n",
+       "│ 3. What is the patient’s age, sex, and any major risk factors (diabetes, hypertension, smoking, family history  │\n",
+       "│ of early CVD, etc.)?                                                                                            │\n",
+       "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
        "
\n"
       ],
       "text/plain": [
-       "\u001b[30;47m╭─\u001b[0m\u001b[30;47m─────────────────────────────────────\u001b[0m\u001b[30;47m \u001b[0m\u001b[1;30;47mOutput (fine-tuned reasoning model)\u001b[0m\u001b[30;47m \u001b[0m\u001b[30;47m─────────────────────────────────────\u001b[0m\u001b[30;47m─╮\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mLet’s start by getting a clearer picture of your bruising and overall health. I’ll ask you a few questions—your\u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47manswers will help me decide whether we need to check for a bleeding disorder or consider other causes:\u001b[0m\u001b[47m         \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m1. Onset & Frequency  \u001b[0m\u001b[47m                                                                                         \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • When did you first notice the bruising?  \u001b[0m\u001b[47m                                                                   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • How many new bruises do you get in a week?  \u001b[0m\u001b[47m                                                                \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m2. Trigger & Distribution  \u001b[0m\u001b[47m                                                                                    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • Do these bruises come up after bumps or minor knocks, or do they appear seemingly out of nowhere?  \u001b[0m\u001b[47m         \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • Are they mostly on pressure points (shins, forearms) or also on areas you wouldn’t expect to hit?  \u001b[0m\u001b[47m         \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m3. Bruise Characteristics  \u001b[0m\u001b[47m                                                                                    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • How large are they, on average?  \u001b[0m\u001b[47m                                                                           \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • Do they change color normally (purple → green → yellow) and heal in a few weeks?  \u001b[0m\u001b[47m                          \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m4. Other Bleeding Symptoms  \u001b[0m\u001b[47m                                                                                   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • Any easy bleeding of gums, frequent nosebleeds, or heavy periods (if applicable)?  \u001b[0m\u001b[47m                         \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • Any blood in your urine or stool?  \u001b[0m\u001b[47m                                                                         \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m5. Medications & Supplements  \u001b[0m\u001b[47m                                                                                 \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • Are you taking any blood thinners (warfarin, heparin, DOACs), aspirin, NSAIDs, or herbal supplements (e.g., \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mfish oil, ginkgo)?  \u001b[0m\u001b[47m                                                                                           \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m6. Medical & Family History  \u001b[0m\u001b[47m                                                                                  \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • Do you have liver disease, kidney disease, or a history of malignancy?  \u001b[0m\u001b[47m                                    \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • Any family history of easy bruising or known bleeding disorders?  \u001b[0m\u001b[47m                                          \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m7. Systemic Symptoms  \u001b[0m\u001b[47m                                                                                         \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47m • Have you noticed fatigue, weight loss, fevers, or night sweats?  \u001b[0m\u001b[47m                                           \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[47m                                                                                                               \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47mOnce I have this information, we can decide whether to check a platelet count, coagulation studies (PT/INR, \u001b[0m\u001b[47m   \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m│\u001b[0m\u001b[47m \u001b[0m\u001b[35;47maPTT), and possibly von Willebrand factor levels. Let me know what you’ve observed.\u001b[0m\u001b[47m                            \u001b[0m\u001b[47m \u001b[0m\u001b[30;47m│\u001b[0m\n",
-       "\u001b[30;47m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
+       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;32mOutput (fine-tuned model)\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mCould you help me narrow this down so I can give you the most relevant recommendation? Specifically:\u001b[0m            \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m1. Which guideline or region are you interested in (for example, USPSTF in the US, ACC/AHA, Canadian, European,\u001b[0m \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35metc.)?  \u001b[0m                                                                                                        \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m2. Are we talking about primary‐prevention screening in an asymptomatic adult, or secondary‑prevention \u001b[0m         \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mmonitoring in someone with known cardiovascular disease?  \u001b[0m                                                      \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35m3. What is the patient’s age, sex, and any major risk factors (diabetes, hypertension, smoking, family history \u001b[0m \u001b[34m│\u001b[0m\n",
+       "\u001b[34m│\u001b[0m \u001b[1;35mof early CVD, etc.)?\u001b[0m                                                                                            \u001b[34m│\u001b[0m\n",
+       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
       ]
      },
      "metadata": {},
@@ -1236,37 +1129,39 @@
    "source": [
     "console = Console()\n",
     "\n",
-    "for test_datapoint in test_data:\n",
-    "    console.print(Panel(\n",
-    "        Text(test_datapoint['messages'][0]['content'], style=\"black\"),\n",
-    "        title=\"[bold black]Input[/bold black]\",\n",
-    "        border_style=\"black\",\n",
-    "        style=\"on white\"\n",
-    "    ))\n",
+    "for item in run_items.to_dict()['data'][:3]:\n",
+    "    input_text = item['datasource_item']['messages'][0]['content']\n",
+    "    output_text = item['datasource_item']['completion'][0]['content']\n",
+    "    sample_text = item['sample']['output'][0]['content']\n",
     "    \n",
     "    console.print(Panel(\n",
-    "        Text(test_datapoint['completion'][0]['content'], style=\"blue\"),\n",
-    "        title=\"[bold black]Output (original model completion)[/bold black]\",\n",
-    "        border_style=\"black\",\n",
-    "        style=\"on white\"\n",
+    "        Text(input_text, style=\"bold cyan\"),\n",
+    "        title=\"[bold green]Input[/bold green]\",\n",
+    "        border_style=\"blue\"\n",
     "    ))\n",
-    "\n",
+    "    \n",
     "    console.print(Panel(\n",
-    "        Text(test_datapoint['base_response'], style=\"dark_green\"),\n",
-    "        title=\"[bold black]Output (base reasoning model)[/bold black]\",\n",
-    "        border_style=\"black\",\n",
-    "        style=\"on white\"\n",
+    "        Text(output_text, style=\"bold yellow\"),\n",
+    "        title=\"[bold green]Output (original model)[/bold green]\",\n",
+    "        border_style=\"blue\"\n",
     "    ))\n",
     "    \n",
     "    console.print(Panel(\n",
-    "        Text(test_datapoint['finetuned_response'], style=\"magenta\"),\n",
-    "        title=\"[bold black]Output (fine-tuned reasoning model)[/bold black]\",\n",
-    "        border_style=\"black\",\n",
-    "        style=\"on white\"\n",
+    "        Text(sample_text, style=\"bold magenta\"),\n",
+    "        title=\"[bold green]Output (fine-tuned model)[/bold green]\",\n",
+    "        border_style=\"blue\"\n",
     "    ))\n",
     "    \n",
     "    console.print(\"\\n\" + \"-\" * 80 + \"\\n\")"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7652f842",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -1285,7 +1180,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.12.9"
+   "version": "3.11.8"
   }
  },
  "nbformat": 4,
diff --git a/examples/gpt-5-codex_prompting_guide.ipynb b/examples/gpt-5-codex_prompting_guide.ipynb
deleted file mode 100644
index 32ebf9b71e..0000000000
--- a/examples/gpt-5-codex_prompting_guide.ipynb
+++ /dev/null
@@ -1,182 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "fefdd2b1",
-   "metadata": {},
-   "source": [
-    "## GPT-5-Codex Prompting Guide\n",
-    "Important details about `GPT-5-Codex` and this guide:\n",
-    "- This model is not a drop-in replacement for GPT-5, as it requires significantly different prompting.\n",
-    "- This model is only supported with the Responses API and does not support the verbosity parameter.\n",
-    "- This guide is meant for API users of `GPT-5-Codex` and creating developer prompts, not for Codex users, if you are a Codex user refer to this [prompting guide](https://developers.openai.com/codex/prompting)\n",
-    "\n",
-    "`GPT-5-Codex` is a new version of GPT‑5 further optimized for agentic and interactive coding tasks. GPT‑5-Codex was trained with a focus on real-world software engineering work; it’s equally proficient at quick, interactive sessions and at independently powering through long, complex tasks. The model builds on GPT-5’s strong coding abilities with additional improvements such as:\n",
-    "- **Improved steerability:** `GPT-5-Codex` delivers higher-quality code on complex engineering tasks like features, tests, debugging, refactors, and reviews without lengthy instructions.\n",
-    "- **Adaptive reasoning level:** `GPT-5-Codex` adjusts its reasoning time to task complexity. It’s snappy in interactive sessions and able to work independently for multiple hours.\n",
-    "- **Excellent at code review:** `GPT-5-Codex` is trained to conduct code reviews, navigating codebases and running code and tests to validate correctness.\n",
-    "\n",
-    "`GPT-5-Codex` is purpose-built for Codex CLI, the Codex IDE extension, the Codex cloud environment, and working in GitHub, and also supports versatile tool use. We recommend using `GPT-5-Codex` only for agentic and interactive coding use cases.\n",
-    "\n",
-    "Because the model is trained specifically for coding, many best practices you once had to prompt into general purpose models are built in, and over prompting can reduce quality. \n",
-    "\n",
-    "The core prompting principle for `GPT-5-Codex` is **“less is more.”**, this includes:\n",
-    "1. Start with a minimal prompt inspired by the Codex CLI system prompt, then add only the essential guidance you truly need.\n",
-    "2. Remove any prompting for preambles, because the model does not support them. Asking for preambles will lead to the model stopping early before completing the task.\n",
-    "3. Reduce the number of tools to only the a terminal tool, and apply_patch.\n",
-    "4. Make tool descriptions as concise as possible by removing unnecessary details.\n",
-    "\n",
-    "\n",
-    "## Codex CLI Prompt\n",
-    "Below is the full Codex CLI developer message, which you can use as the reference implementation for prompting `GPT-5-Codex`. Compared with the GPT-5 developer message, it uses about 40% as many tokens, reinforcing that minimal prompting is ideal for this model.\n",
-    "\n",
-    "\n",
-    "\n",
-    "Here is a link to the [GPT-5-Codex Prompt](https://github.com/openai/codex/blob/main/codex-rs/core/gpt_5_codex_prompt.md) within Codex CLI as well as the [GPT-5 prompt](https://github.com/openai/codex/blob/main/codex-rs/core/prompt.md). As a point of comparison you can see the `GPT-5-Codex` prompt is much shorter than GPT-5 and we recommend following the same pattern. \n",
-    "```\n",
-    "You are Codex, based on GPT-5. You are running as a coding agent in the Codex CLI on a user's computer.\n",
-    "\n",
-    "## General\n",
-    "\n",
-    "- The arguments to `shell` will be passed to execvp(). Most terminal commands should be prefixed with [\"bash\", \"-lc\"].\n",
-    "- Always set the `workdir` param when using the shell function. Do not use `cd` unless absolutely necessary.\n",
-    "- When searching for text or files, prefer using `rg` or `rg --files` respectively because `rg` is much faster than alternatives like `grep`. (If the `rg` command is not found, then use alternatives.)\n",
-    "\n",
-    "## Editing constraints\n",
-    "\n",
-    "- Default to ASCII when editing or creating files. Only introduce non-ASCII or other Unicode characters when there is a clear justification and the file already uses them.\n",
-    "- Add succinct code comments that explain what is going on if code is not self-explanatory. You should not add comments like \"Assigns the value to the variable\", but a brief comment might be useful ahead of a complex code block that the user would otherwise have to spend time parsing out. Usage of these comments should be rare.\n",
-    "- You may be in a dirty git worktree.\n",
-    "    * NEVER revert existing changes you did not make unless explicitly requested, since these changes were made by the user.\n",
-    "    * If asked to make a commit or code edits and there are unrelated changes to your work or changes that you didn't make in those files, don't revert those changes.\n",
-    "    * If the changes are in files you've touched recently, you should read carefully and understand how you can work with the changes rather than reverting them.\n",
-    "    * If the changes are in unrelated files, just ignore them and don't revert them.\n",
-    "- While you are working, you might notice unexpected changes that you didn't make. If this happens, STOP IMMEDIATELY and ask the user how they would like to proceed.\n",
-    "\n",
-    "## Plan tool\n",
-    "\n",
-    "When using the planning tool:\n",
-    "- Skip using the planning tool for straightforward tasks (roughly the easiest 25%).\n",
-    "- Do not make single-step plans.\n",
-    "- When you made a plan, update it after having performed one of the sub-tasks that you shared on the plan.\n",
-    "\n",
-    "## Codex CLI harness, sandboxing, and approvals\n",
-    "\n",
-    "The Codex CLI harness supports several different configurations for sandboxing and escalation approvals that the user can choose from.\n",
-    "\n",
-    "Filesystem sandboxing defines which files can be read or written. The options for `sandbox_mode` are:\n",
-    "- **read-only**: The sandbox only permits reading files.\n",
-    "- **workspace-write**: The sandbox permits reading files, and editing files in `cwd` and `writable_roots`. Editing files in other directories requires approval.\n",
-    "- **danger-full-access**: No filesystem sandboxing - all commands are permitted.\n",
-    "\n",
-    "Network sandboxing defines whether network can be accessed without approval. Options for `network_access` are:\n",
-    "- **restricted**: Requires approval\n",
-    "- **enabled**: No approval needed\n",
-    "\n",
-    "Approvals are your mechanism to get user consent to run shell commands without the sandbox. Possible configuration options for `approval_policy` are\n",
-    "- **untrusted**: The harness will escalate most commands for user approval, apart from a limited allowlist of safe \"read\" commands.\n",
-    "- **on-failure**: The harness will allow all commands to run in the sandbox (if enabled), and failures will be escalated to the user for approval to run again without the sandbox.\n",
-    "- **on-request**: Commands will be run in the sandbox by default, and you can specify in your tool call if you want to escalate a command to run without sandboxing. (Note that this mode is not always available. If it is, you'll see parameters for it in the `shell` command description.)\n",
-    "- **never**: This is a non-interactive mode where you may NEVER ask the user for approval to run commands. Instead, you must always persist and work around constraints to solve the task for the user. You MUST do your utmost best to finish the task and validate your work before yielding. If this mode is paired with `danger-full-access`, take advantage of it to deliver the best outcome for the user. Further, in this mode, your default testing philosophy is overridden: Even if you don't see local patterns for testing, you may add tests and scripts to validate your work. Just remove them before yielding.\n",
-    "\n",
-    "When you are running with `approval_policy == on-request`, and sandboxing enabled, here are scenarios where you'll need to request approval:\n",
-    "- You need to run a command that writes to a directory that requires it (e.g. running tests that write to /var)\n",
-    "- You need to run a GUI app (e.g., open/xdg-open/osascript) to open browsers or files.\n",
-    "- You are running sandboxed and need to run a command that requires network access (e.g. installing packages)\n",
-    "- If you run a command that is important to solving the user's query, but it fails because of sandboxing, rerun the command with approval. ALWAYS proceed to use the `with_escalated_permissions` and `justification` parameters - do not message the user before requesting approval for the command.\n",
-    "- You are about to take a potentially destructive action such as an `rm` or `git reset` that the user did not explicitly ask for\n",
-    "- (for all of these, you should weigh alternative paths that do not require approval)\n",
-    "\n",
-    "When `sandbox_mode` is set to read-only, you'll need to request approval for any command that isn't a read.\n",
-    "\n",
-    "You will be told what filesystem sandboxing, network sandboxing, and approval mode are active in a developer or user message. If you are not told about this, assume that you are running with workspace-write, network sandboxing enabled, and approval on-failure.\n",
-    "\n",
-    "Although they introduce friction to the user because your work is paused until the user responds, you should leverage them when necessary to accomplish important work. If the completing the task requires escalated permissions, Do not let these settings or the sandbox deter you from attempting to accomplish the user's task unless it is set to \"never\", in which case never ask for approvals.\n",
-    "\n",
-    "When requesting approval to execute a command that will require escalated privileges:\n",
-    "  - Provide the `with_escalated_permissions` parameter with the boolean value true\n",
-    "  - Include a short, 1 sentence explanation for why you need to enable `with_escalated_permissions` in the justification parameter\n",
-    "\n",
-    "## Special user requests\n",
-    "\n",
-    "- If the user makes a simple request (such as asking for the time) which you can fulfill by running a terminal command (such as `date`), you should do so.\n",
-    "- If the user asks for a \"review\", default to a code review mindset: prioritise identifying bugs, risks, behavioural regressions, and missing tests. Findings must be the primary focus of the response - keep summaries or overviews brief and only after enumerating the issues. Present findings first (ordered by severity with file/line references), follow with open questions or assumptions, and offer a change-summary only as a secondary detail. If no findings are discovered, state that explicitly and mention any residual risks or testing gaps.\n",
-    "\n",
-    "## Presenting your work and final message\n",
-    "\n",
-    "You are producing plain text that will later be styled by the CLI. Follow these rules exactly. Formatting should make results easy to scan, but not feel mechanical. Use judgment to decide how much structure adds value.\n",
-    "\n",
-    "- Default: be very concise; friendly coding teammate tone.\n",
-    "- Ask only when needed; suggest ideas; mirror the user's style.\n",
-    "- For substantial work, summarize clearly; follow final‑answer formatting.\n",
-    "- Skip heavy formatting for simple confirmations.\n",
-    "- Don't dump large files you've written; reference paths only.\n",
-    "- No \"save/copy this file\" - User is on the same machine.\n",
-    "- Offer logical next steps (tests, commits, build) briefly; add verify steps if you couldn't do something.\n",
-    "- For code changes:\n",
-    "  * Lead with a quick explanation of the change, and then give more details on the context covering where and why a change was made. Do not start this explanation with \"summary\", just jump right in.\n",
-    "  * If there are natural next steps the user may want to take, suggest them at the end of your response. Do not make suggestions if there are no natural next steps.\n",
-    "  * When suggesting multiple options, use numeric lists for the suggestions so the user can quickly respond with a single number.\n",
-    "- The user does not command execution outputs. When asked to show the output of a command (e.g. `git show`), relay the important details in your answer or summarize the key lines so the user understands the result.\n",
-    "\n",
-    "### Final answer structure and style guidelines\n",
-    "\n",
-    "- Plain text; CLI handles styling. Use structure only when it helps scanability.\n",
-    "- Headers: optional; short Title Case (1-3 words) wrapped in **…**; no blank line before the first bullet; add only if they truly help.\n",
-    "- Bullets: use - ; merge related points; keep to one line when possible; 4–6 per list ordered by importance; keep phrasing consistent.\n",
-    "- Monospace: backticks for commands/paths/env vars/code ids and inline examples; use for literal keyword bullets; never combine with **.\n",
-    "- Code samples or multi-line snippets should be wrapped in fenced code blocks; add a language hint whenever obvious.\n",
-    "- Structure: group related bullets; order sections general → specific → supporting; for subsections, start with a bolded keyword bullet, then items; match complexity to the task.\n",
-    "- Tone: collaborative, concise, factual; present tense, active voice; self‑contained; no \"above/below\"; parallel wording.\n",
-    "- Don'ts: no nested bullets/hierarchies; no ANSI codes; don't cram unrelated keywords; keep keyword lists short—wrap/reformat if long; avoid naming formatting styles in answers.\n",
-    "- Adaptation: code explanations → precise, structured with code refs; simple tasks → lead with outcome; big changes → logical walkthrough + rationale + next actions; casual one-offs → plain sentences, no headers/bullets.\n",
-    "- File References: When referencing files in your response, make sure to include the relevant start line and always follow the below rules:\n",
-    "  * Use inline code to make file paths clickable.\n",
-    "  * Each reference should have a stand alone path. Even if it's the same file.\n",
-    "  * Accepted: absolute, workspace‑relative, a/ or b/ diff prefixes, or bare filename/suffix.\n",
-    "  * Line/column (1‑based, optional): :line[:column] or #Lline[Ccolumn] (column defaults to 1).\n",
-    "  * Do not use URIs like file://, vscode://, or https://.\n",
-    "  * Do not provide range of lines\n",
-    "  * Examples: src/app.ts, src/app.ts:42, b/server/index.js#L10, C:\\repo\\project\\main.rs:12:5\n",
-    "```\n",
-    "#### Apply Patch\n",
-    "As shared previously in the `GPT-5` prompting guide, [here](https://github.com/openai/openai-cookbook/tree/main/examples/gpt-5/apply_patch.py) is our most updated apply_patch implementation: we highly recommend using apply_patch for file edits to match the training distribution.\n",
-    "\n",
-    "## Anti-Prompting\n",
-    "As noted above, because `GPT-5-Codex` was trained for optimal agentic coding, prompt tuning will more often mean removing guidance than adding it. Below are aspects you may not need to steer.\n",
-    "\n",
-    "#### Adaptive Reasoning\n",
-    "Adaptive reasoning is now the default in `GPT-5-Codex`. In the past, you might have prompted models to “think harder” or “respond quickly” based on task difficulty. `GPT-5-Codex` adjusts automatically: for a question like “How do I undo the last commit but keep all changes staged?”, it responds quickly without extra steering. For more complex coding tasks, it takes the time it needs and uses tools as appropriate.\n",
-    "\n",
-    "#### Planning\n",
-    "`GPT-5-Codex` was trained for a wide variety of coding tasks from long-running agentic tasks to shorter interactive coding tasks, so the model has a collaborative personality by default. When you kick off an agentic task, the model will build a detailed plan and keep you updated as it progresses. Codex CLI includes a planning tool, and the model is trained to use it throughout its agentic rollout, so if you provide a planning tool as well, the model can leverage it while coding.\n",
-    "The [”Planning” section of the GPT-5 dev message in Codex CLI](https://github.com/openai/codex/blob/main/codex-rs/core/prompt.md?plain=1#L52-L122) is no longer needed in `GPT-5-Codex`, as the model is trained to produce high-quality plans.\n",
-    "\n",
-    "#### Preambles\n",
-    "**`GPT-5-Codex` does not emit preambles!** Prompting and asking for it will likely result in the model stopping early. Instead, we have a custom summarizer that produces detailed summaries only when appropriate so you can render them inline.\n",
-    "\n",
-    "#### Frontend\n",
-    "`GPT-5-Codex` defaults to strong aesthetics and modern frontend best practices. If you have preferred libraries or frameworks, steer the model by adding short sections that spell them out, such as:\n",
-    "\n",
-    "```\n",
-    "Frontend Guidance\n",
-    "Use the following libraries unless the user or repo specifies otherwise:\n",
-    "Framework: React + TypeScript\n",
-    "Styling: Tailwind CSS\n",
-    "Components: shadcn/ui\n",
-    "Icons: lucide-react\n",
-    "Animation: Framer Motion\n",
-    "Charts: Recharts\n",
-    "Fonts: San Serif, Inter, Geist, Mona Sans, IBM Plex Sans, Manrope\n",
-    "```\n"
-   ]
-  }
- ],
- "metadata": {
-  "language_info": {
-   "name": "python"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/examples/partners/model_selection_guide/model_selection_guide.ipynb b/examples/partners/model_selection_guide/model_selection_guide.ipynb
index 8b4b9e6604..19416b2e81 100644
--- a/examples/partners/model_selection_guide/model_selection_guide.ipynb
+++ b/examples/partners/model_selection_guide/model_selection_guide.ipynb
@@ -2425,7 +2425,7 @@
     "\n",
     "- **[Orchestrating Agents: Routines and Handoffs](https://cookbook.openai.com/examples/orchestrating_agents)** Structuring multi-agent workflows with routines and handoffs, relevant to the ideation→ranking→critique pipeline.\n",
     "\n",
-    "- **[GPT-4.1 Prompting Guide](https://cookbook.openai.com/examples/gpt4-1_prompting_guide)** Advanced prompting, tool use, and task decomposition for improved accuracy in critique and safety reviews.\n",
+    "- **[GPT-4.1 Prompting Guide](https://cookbook.openai.com/examples/prompting/gpt4-1_prompting_guide)** Advanced prompting, tool use, and task decomposition for improved accuracy in critique and safety reviews.\n",
     "\n",
     "- **[Structured Outputs for Multi-Agent Systems](https://cookbook.openai.com/examples/structured_outputs_multi_agent)** Enforcing consistent JSON outputs with schema validation for agent interoperability.\n",
     "\n",
@@ -3219,7 +3219,7 @@
     "- [Data Extraction and Transformation](https://cookbook.openai.com/examples/data_extraction_transformation)\n",
     "\n",
     "### Prompting & Model Selection\n",
-    "- [GPT-4.1 Prompting Guide](https://cookbook.openai.com/examples/gpt4-1_prompting_guide)\n",
+    "- [GPT-4.1 Prompting Guide](https://cookbook.openai.com/examples/prompting/gpt4-1_prompting_guide)\n",
     "- [Prompt Engineering Best Practices](https://platform.openai.com/docs/guides/prompt-engineering)\n",
     "\n",
     "### Evaluation & Deployment\n",
diff --git a/examples/Enhance_your_prompts_with_meta_prompting.ipynb b/examples/prompting/Enhance_your_prompts_with_meta_prompting.ipynb
similarity index 100%
rename from examples/Enhance_your_prompts_with_meta_prompting.ipynb
rename to examples/prompting/Enhance_your_prompts_with_meta_prompting.ipynb
diff --git a/examples/Optimize_Prompts.ipynb b/examples/prompting/Optimize_Prompts.ipynb
similarity index 100%
rename from examples/Optimize_Prompts.ipynb
rename to examples/prompting/Optimize_Prompts.ipynb
diff --git a/examples/Prompt_migration_guide.ipynb b/examples/prompting/Prompt_migration_guide.ipynb
similarity index 100%
rename from examples/Prompt_migration_guide.ipynb
rename to examples/prompting/Prompt_migration_guide.ipynb
diff --git a/examples/prompting/README.md b/examples/prompting/README.md
new file mode 100644
index 0000000000..c530981fc1
--- /dev/null
+++ b/examples/prompting/README.md
@@ -0,0 +1,59 @@
+# Prompting: Guides & Examples
+
+This directory consolidates prompting-related guides, examples, and reusable prompt assets from across the Cookbook. It’s a single place to learn core prompting patterns, optimize prompts, and discover application-specific prompt sets.
+
+## Why this exists
+Effective prompting solves a large share of practical model issues. The right prompt is as important as parameters like `temperature` or `reasoning_effort`. Centralizing examples and references makes them easier to find, reuse, and maintain.
+
+## Start here (recommended path)
+1. **GPT-4.1 Prompting Guide** → techniques, structure, and patterns:  
+   [`gpt4-1_prompting_guide.ipynb`](./gpt4-1_prompting_guide.ipynb)
+2. **Prompt engineering best practices** (reference):  
+   
+3. **Orchestrating agents & handoffs** (for multi-agent apps):  
+   `../orchestrating_agents` (see the top-level Examples index)
+4. **Structured outputs** (JSON schemas, validation):  
+   `../structured_outputs_multi_agent`
+
+> Tip: Keep prompts short, specific, and testable. Add minimal examples, define outputs precisely, and prefer explicit instructions over implications.
+
+## Contents
+
+### Core Guides
+- **GPT-4.1 Prompting Guide** — system prompts, tool use, decomposition, evaluation  
+  [`gpt4-1_prompting_guide.ipynb`](./gpt4-1_prompting_guide.ipynb)
+- **Realtime prompting guide** — working with the Realtime API  
+  [`Realtime_prompting_guide.ipynb`](./Realtime_prompting_guide.ipynb)
+- **Whisper prompting guide** — task hints and formatting for speech recognition  
+  [`Whisper_prompting_guide.ipynb`](./Whisper_prompting_guide.ipynb)
+
+### Prompt Optimization
+- **Optimize Prompts** — automated checks & fixes for common prompt issues  
+  [`Optimize_Prompts.ipynb`](./Optimize_Prompts.ipynb)
+- **Enhance your prompts with meta-prompting** — programmatic refinement strategies  
+  [`Enhance_your_prompts_with_meta_prompting.ipynb`](./Enhance_your_prompts_with_meta_prompting.ipynb)
+- **Prompt migration guide** — safely updating existing prompts across changes  
+  [`Prompt_migration_guide.ipynb`](./Prompt_migration_guide.ipynb)
+
+### Agent & App Prompts
+- **Multi-agent portfolio collaboration prompts** — reusable prompt set for agent roles  
+  [`../agents_sdk/multi-agent-portfolio-collaboration/prompts/`](../agents_sdk/multi-agent-portfolio-collaboration/prompts/)
+
+### Supporting Resources
+- OpenAI Prompt Engineering (plain-text reference)  
+  [`../data/oai_docs/prompt-engineering.txt`](../data/oai_docs/prompt-engineering.txt)
+
+## Usage pattern
+
+1. **Draft** a minimal instruction with explicit output shape (e.g., JSON schema).
+2. **Ground** with constraints (tone, audience, knowledge limits) and a tiny example if needed.
+3. **Test** with real inputs; watch for ambiguity and output drift.
+4. **Evaluate** with checks (format validation, assertions).
+5. **Iterate**: shorten, remove redundant rules, and pin “must-haves”.
+
+## Contributing
+
+- Keep notebooks **runnable end-to-end** (no hidden cell state).
+- Prefer **relative links** within `examples/`, so both GitHub and the site render cleanly.
+- When adding new files here, **update** `registry.yaml` so content appears on the site.
+- If you introduce a new subfolder of prompts, include a short `README.md` explaining scope and usage.
diff --git a/examples/Realtime_prompting_guide.ipynb b/examples/prompting/Realtime_prompting_guide.ipynb
similarity index 100%
rename from examples/Realtime_prompting_guide.ipynb
rename to examples/prompting/Realtime_prompting_guide.ipynb
diff --git a/examples/Unit_test_writing_using_a_multi-step_prompt.ipynb b/examples/prompting/Unit_test_writing_using_a_multi-step_prompt.ipynb
similarity index 100%
rename from examples/Unit_test_writing_using_a_multi-step_prompt.ipynb
rename to examples/prompting/Unit_test_writing_using_a_multi-step_prompt.ipynb
diff --git a/examples/Unit_test_writing_using_a_multi-step_prompt_with_older_completions_API.ipynb b/examples/prompting/Unit_test_writing_using_a_multi-step_prompt_with_older_completions_API.ipynb
similarity index 100%
rename from examples/Unit_test_writing_using_a_multi-step_prompt_with_older_completions_API.ipynb
rename to examples/prompting/Unit_test_writing_using_a_multi-step_prompt_with_older_completions_API.ipynb
diff --git a/examples/Whisper_prompting_guide.ipynb b/examples/prompting/Whisper_prompting_guide.ipynb
similarity index 100%
rename from examples/Whisper_prompting_guide.ipynb
rename to examples/prompting/Whisper_prompting_guide.ipynb
diff --git a/examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/code_interpreter.md b/examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/code_interpreter.md
similarity index 100%
rename from examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/code_interpreter.md
rename to examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/code_interpreter.md
diff --git a/examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/editor_base.md b/examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/editor_base.md
similarity index 100%
rename from examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/editor_base.md
rename to examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/editor_base.md
diff --git a/examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/fundamental_base.md b/examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/fundamental_base.md
similarity index 100%
rename from examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/fundamental_base.md
rename to examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/fundamental_base.md
diff --git a/examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/macro_base.md b/examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/macro_base.md
similarity index 100%
rename from examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/macro_base.md
rename to examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/macro_base.md
diff --git a/examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/pm_base.md b/examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/pm_base.md
similarity index 100%
rename from examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/pm_base.md
rename to examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/pm_base.md
diff --git a/examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/quant_base.md b/examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/quant_base.md
similarity index 100%
rename from examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/quant_base.md
rename to examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/quant_base.md
diff --git a/examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/tool_retry_prompt.md b/examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/tool_retry_prompt.md
similarity index 100%
rename from examples/agents_sdk/multi-agent-portfolio-collaboration/prompts/tool_retry_prompt.md
rename to examples/prompting/agents_sdk/multi-agent-portfolio-collaboration/prompts/tool_retry_prompt.md
diff --git a/examples/data/oai_docs/prompt-engineering.txt b/examples/prompting/data/oai_docs/prompt-engineering.txt
similarity index 100%
rename from examples/data/oai_docs/prompt-engineering.txt
rename to examples/prompting/data/oai_docs/prompt-engineering.txt
diff --git a/examples/gpt-5/gpt-5_prompting_guide.ipynb b/examples/prompting/gpt-5/gpt-5_prompting_guide.ipynb
similarity index 100%
rename from examples/gpt-5/gpt-5_prompting_guide.ipynb
rename to examples/prompting/gpt-5/gpt-5_prompting_guide.ipynb
diff --git a/examples/gpt-5/prompt-optimization-cookbook.ipynb b/examples/prompting/gpt-5/prompt-optimization-cookbook.ipynb
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook.ipynb
rename to examples/prompting/gpt-5/prompt-optimization-cookbook.ipynb
diff --git a/examples/gpt-5/prompt-optimization-cookbook/llm_as_judge.txt b/examples/prompting/gpt-5/prompt-optimization-cookbook/llm_as_judge.txt
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/llm_as_judge.txt
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/llm_as_judge.txt
diff --git a/examples/gpt-5/prompt-optimization-cookbook/requirements.txt b/examples/prompting/gpt-5/prompt-optimization-cookbook/requirements.txt
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/requirements.txt
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/requirements.txt
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_failsafeqa_baseline.csv b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_failsafeqa_baseline.csv
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_failsafeqa_baseline.csv
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_failsafeqa_baseline.csv
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_failsafeqa_optimized.csv b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_failsafeqa_optimized.csv
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_failsafeqa_optimized.csv
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_failsafeqa_optimized.csv
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/judgement_summary.csv b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/judgement_summary.csv
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/judgement_summary.csv
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/judgement_summary.csv
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_01.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_01.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_01.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_01.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_02.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_02.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_02.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_02.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_03.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_03.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_03.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_03.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_04.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_04.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_04.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_04.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_05.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_05.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_05.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_05.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_06.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_06.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_06.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_06.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_07.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_07.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_07.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_07.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_08.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_08.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_08.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_08.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_09.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_09.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_09.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_09.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_10.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_10.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_10.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_10.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_11.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_11.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_11.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_11.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_12.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_12.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_12.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_12.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_13.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_13.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_13.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_13.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_14.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_14.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_14.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_14.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_15.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_15.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_15.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_15.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_16.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_16.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_16.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_16.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_17.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_17.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_17.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_17.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_18.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_18.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_18.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_18.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_19.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_19.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_19.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_19.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_20.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_20.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_20.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_20.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_21.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_21.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_21.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_21.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_22.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_22.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_22.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_22.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_23.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_23.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_23.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_23.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_24.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_24.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_24.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_24.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_25.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_25.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_25.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_25.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_26.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_26.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_26.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_26.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_27.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_27.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_27.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_27.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_28.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_28.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_28.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_28.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_29.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_29.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_29.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_29.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_30.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_30.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_30.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_baseline/run_30.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/judgement_summary.csv b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/judgement_summary.csv
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/judgement_summary.csv
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/judgement_summary.csv
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_01.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_01.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_01.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_01.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_02.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_02.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_02.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_02.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_03.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_03.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_03.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_03.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_04.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_04.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_04.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_04.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_05.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_05.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_05.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_05.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_06.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_06.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_06.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_06.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_07.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_07.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_07.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_07.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_08.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_08.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_08.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_08.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_09.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_09.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_09.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_09.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_10.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_10.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_10.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_10.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_11.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_11.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_11.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_11.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_12.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_12.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_12.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_12.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_13.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_13.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_13.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_13.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_14.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_14.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_14.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_14.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_15.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_15.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_15.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_15.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_16.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_16.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_16.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_16.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_17.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_17.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_17.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_17.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_18.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_18.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_18.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_18.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_19.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_19.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_19.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_19.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_20.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_20.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_20.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_20.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_21.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_21.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_21.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_21.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_22.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_22.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_22.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_22.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_23.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_23.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_23.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_23.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_24.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_24.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_24.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_24.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_25.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_25.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_25.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_25.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_26.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_26.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_26.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_26.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_27.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_27.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_27.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_27.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_28.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_28.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_28.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_28.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_29.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_29.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_29.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_29.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_30.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_30.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_30.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_llm_as_judge_optimized/run_30.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_01.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_01.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_01.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_01.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_02.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_02.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_02.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_02.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_03.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_03.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_03.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_03.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_04.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_04.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_04.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_04.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_05.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_05.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_05.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_05.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_06.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_06.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_06.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_06.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_07.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_07.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_07.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_07.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_08.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_08.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_08.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_08.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_09.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_09.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_09.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_09.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_10.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_10.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_10.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_10.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_11.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_11.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_11.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_11.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_12.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_12.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_12.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_12.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_13.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_13.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_13.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_13.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_14.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_14.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_14.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_14.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_15.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_15.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_15.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_15.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_16.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_16.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_16.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_16.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_17.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_17.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_17.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_17.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_18.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_18.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_18.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_18.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_19.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_19.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_19.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_19.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_20.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_20.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_20.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_20.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_21.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_21.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_21.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_21.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_22.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_22.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_22.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_22.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_23.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_23.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_23.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_23.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_24.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_24.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_24.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_24.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_25.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_25.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_25.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_25.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_26.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_26.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_26.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_26.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_27.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_27.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_27.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_27.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_28.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_28.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_28.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_28.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_29.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_29.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_29.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_29.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_30.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_30.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_30.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_30.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline.csv b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline.csv
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline.csv
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline.csv
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline_summary.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline_summary.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline_summary.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline_summary.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline_summary.txt b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline_summary.txt
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline_summary.txt
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_baseline/run_results_topk_baseline_summary.txt
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_01.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_01.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_01.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_01.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_02.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_02.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_02.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_02.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_03.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_03.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_03.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_03.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_04.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_04.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_04.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_04.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_05.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_05.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_05.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_05.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_06.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_06.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_06.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_06.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_07.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_07.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_07.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_07.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_08.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_08.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_08.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_08.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_09.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_09.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_09.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_09.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_10.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_10.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_10.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_10.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_11.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_11.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_11.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_11.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_12.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_12.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_12.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_12.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_13.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_13.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_13.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_13.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_14.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_14.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_14.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_14.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_15.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_15.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_15.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_15.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_16.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_16.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_16.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_16.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_17.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_17.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_17.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_17.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_18.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_18.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_18.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_18.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_19.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_19.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_19.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_19.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_20.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_20.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_20.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_20.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_21.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_21.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_21.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_21.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_22.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_22.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_22.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_22.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_23.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_23.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_23.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_23.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_24.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_24.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_24.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_24.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_25.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_25.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_25.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_25.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_26.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_26.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_26.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_26.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_27.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_27.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_27.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_27.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_28.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_28.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_28.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_28.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_29.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_29.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_29.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_29.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_30.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_30.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_30.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_30.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized.csv b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized.csv
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized.csv
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized.csv
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized_summary.json b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized_summary.json
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized_summary.json
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized_summary.json
diff --git a/examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized_summary.txt b/examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized_summary.txt
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized_summary.txt
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/results_topk_optimized/run_results_topk_optimized_summary.txt
diff --git a/examples/gpt-5/prompt-optimization-cookbook/run_FailSafeQA.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/run_FailSafeQA.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/run_FailSafeQA.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/run_FailSafeQA.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/scripts/__init__.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/__init__.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/scripts/__init__.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/__init__.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/scripts/gen_baseline.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/gen_baseline.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/scripts/gen_baseline.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/gen_baseline.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/scripts/gen_optimized.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/gen_optimized.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/scripts/gen_optimized.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/gen_optimized.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/scripts/llm_judge.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/llm_judge.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/scripts/llm_judge.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/llm_judge.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/scripts/results_summarizer.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/results_summarizer.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/scripts/results_summarizer.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/results_summarizer.py
diff --git a/examples/gpt-5/prompt-optimization-cookbook/scripts/topk_eval.py b/examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/topk_eval.py
similarity index 100%
rename from examples/gpt-5/prompt-optimization-cookbook/scripts/topk_eval.py
rename to examples/prompting/gpt-5/prompt-optimization-cookbook/scripts/topk_eval.py
diff --git a/examples/gpt4-1_prompting_guide.ipynb b/examples/prompting/gpt4-1_prompting_guide.ipynb
similarity index 100%
rename from examples/gpt4-1_prompting_guide.ipynb
rename to examples/prompting/gpt4-1_prompting_guide.ipynb
diff --git a/examples/o-series/o3o4-mini_prompting_guide.ipynb b/examples/prompting/o-series/o3o4-mini_prompting_guide.ipynb
similarity index 100%
rename from examples/o-series/o3o4-mini_prompting_guide.ipynb
rename to examples/prompting/o-series/o3o4-mini_prompting_guide.ipynb
diff --git a/examples/voice_solutions/one_way_translation_using_realtime_api/src/utils/translation_prompts.js b/examples/prompting/voice_solutions/one_way_translation_using_realtime_api/src/utils/translation_prompts.js
similarity index 100%
rename from examples/voice_solutions/one_way_translation_using_realtime_api/src/utils/translation_prompts.js
rename to examples/prompting/voice_solutions/one_way_translation_using_realtime_api/src/utils/translation_prompts.js
diff --git a/registry.yaml b/registry.yaml
index 3b38ccbe10..80c1033819 100644
--- a/registry.yaml
+++ b/registry.yaml
@@ -4,15 +4,6 @@
 # should build pages for, and indicates metadata such as tags, creation date and
 # authors for each page.
 
-- title: GPT-5-Codex Prompting Guide
-  path: examples/gpt-5-codex_prompting_guide.ipynb
-  date: 2025-09-23
-  authors:
-    - daveleo-openai
-  tags:
-    - gpt-5
-    - codex
-
 - title: GPT-5 Troubleshooting Guide
   path: examples/gpt-5/gpt-5_troubleshooting_guide.ipynb
   date: 2025-09-17
@@ -42,7 +33,7 @@
     - codex
 
 - title: Realtime Prompting Guide
-  path: examples/Realtime_prompting_guide.ipynb
+  path: examples/prompting/Realtime_prompting_guide.ipynb
   date: 2025-08-28
   authors:
     - minh-hoque
@@ -84,7 +75,7 @@
     - gpt-oss-local
 
 - title: GPT-5 Prompt Migration and Improvement Using the New Optimizer
-  path: examples/gpt-5/prompt-optimization-cookbook.ipynb
+  path: examples/prompting/gpt-5/prompt-optimization-cookbook.ipynb
   date: 2025-08-07
   authors:
     - rajpathak-openai
@@ -96,7 +87,7 @@
     - prompt-optimization
 
 - title: GPT-5 prompting guide
-  path: examples/gpt-5/gpt-5_prompting_guide.ipynb
+  path: examples/prompting/gpt-5/gpt-5_prompting_guide.ipynb
   date: 2025-08-07
   authors:
     - anoop-openai
@@ -249,7 +240,7 @@
     - audio
 
 - title: Optimize Prompts
-  path: examples/Optimize_Prompts.ipynb
+  path: examples/prompting/Optimize_Prompts.ipynb
   date: 2025-07-14
   authors:
     - corwin
@@ -271,7 +262,7 @@
     - automation
 
 - title: Prompt Migration Guide
-  path: examples/Prompt_migration_guide.ipynb
+  path: examples/prompting/Prompt_migration_guide.ipynb
   date: 2025-06-26
   authors:
     - minh-hoque
@@ -342,7 +333,7 @@
     - mutli-agent-collaboration
 
 - title: o3/o4-mini Function Calling Guide
-  path: examples/o-series/o3o4-mini_prompting_guide.ipynb
+  path: examples/prompting/o-series/o3o4-mini_prompting_guide.ipynb
   date: 2025-05-26
   authors:
     - billchen-openai
@@ -796,7 +787,7 @@
     - embeddings
 
 - title: Unit test writing using a multi-step prompt
-  path: examples/Unit_test_writing_using_a_multi-step_prompt.ipynb
+  path: examples/prompting/Unit_test_writing_using_a_multi-step_prompt.ipynb
   date: 2022-11-15
   authors:
     - ted-at-openai
@@ -804,8 +795,7 @@
     - completions
 
 - title: Unit test writing using a multi-step prompt with legacy Completions
-  path: >-
-    examples/Unit_test_writing_using_a_multi-step_prompt_with_older_completions_API.ipynb
+  path: examples/prompting/Unit_test_writing_using_a_multi-step_prompt_with_older_completions_API.ipynb
   date: 2023-05-19
   authors:
     - ted-at-openai
@@ -875,7 +865,7 @@
   archived: true
 
 - title: Whisper prompting guide
-  path: examples/Whisper_prompting_guide.ipynb
+  path: examples/prompting/Whisper_prompting_guide.ipynb
   date: 2023-06-27
   authors:
     - prestontuggle
@@ -2194,7 +2184,7 @@
     - audio
 
 - title: Enhance your prompts with meta prompting
-  path: examples/Enhance_your_prompts_with_meta_prompting.ipynb
+  path: examples/prompting/Enhance_your_prompts_with_meta_prompting.ipynb
   date: 2024-10-23
   authors:
     - teomusatoiu
@@ -2377,7 +2367,7 @@
     - chatgpt-productivity
 
 - title: GPT-4.1 Prompting Guide
-  path: examples/gpt4-1_prompting_guide.ipynb
+  path: examples/prompting/gpt4-1_prompting_guide.ipynb
   date: 2025-04-14
   authors:
     - nm-openai
@@ -2530,16 +2520,11 @@
   tags:
     - images
 
-
-- title: Codex CLI to automatically fix CI failures
-  path: examples/codex/codex-cicd.ipynb
-  date: 2025-09-30
+- title: LLMs 101: A Practical Introduction
+  path: articles/openai-cookbook-llms-101.md
+  date: 2025-09-15
   authors:
-    - himadri518
-    - alwell-kevin
+    - paytonison
   tags:
-    - codex
-
-
-
-    
+    - llms
+    - beginners