Skip to content

Commit ea29045

Browse files
mrbullwinklemrbullwinkle
committed
update
1 parent c2f474a commit ea29045

File tree

1 file changed

+181
-54
lines changed

1 file changed

+181
-54
lines changed

‎articles/ai-foundry/openai/how-to/reasoning.md

Lines changed: 181 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ Azure OpenAI reasoning models are designed to tackle reasoning and problem-solvi
6262
| Streaming | ✅ | ✅ | ✅|
6363

6464
<sup>1</sup> Parallel tool calls are not supported when `reasoning_effort` is set to `minimal`<br><br>
65-
<sup>2</sup> Reasoning models will only work with the `max_completion_tokens` parameter. <br><br>
65+
<sup>2</sup> Reasoning models will only work with the `max_completion_tokens` parameter when using the Chat Completions API. Use `max_output_tokens` with the Responses API. <br><br>
6666
<sup>3</sup> The latest reasoning models support system messages to make migration easier. You should not use both a developer message and a system message in the same API request.<br><br>
6767
<sup>4</sup> Access to the chain-of-thought reasoning summary is limited access only for `o3` & `o4-mini`.
6868

@@ -98,7 +98,7 @@ For more information, we also recommend reading OpenAI's [GPT-5 prompting cookbo
9898
| [Reasoning summary](#reasoning-summary) <sup>3</sup> | ✅ | - | ✅ | ✅ | - | - | - |
9999
| Streaming <sup>4</sup> | ✅ | - | ✅ | ✅| ✅ | - | - |
100100

101-
<sup>1</sup> Reasoning models will only work with the `max_completion_tokens` parameter. <br><br>
101+
<sup>1</sup> Reasoning models will only work with the `max_completion_tokens` parameter when using the Chat Completions API. Use `max_output_tokens` with the Responses API.<br><br>
102102
<sup>2</sup> The latest o<sup>&#42;</sup> series model support system messages to make migration easier. When you use a system message with `o4-mini`, `o3`, `o3-mini`, and `o1` it will be treated as a developer message. You should not use both a developer message and a system message in the same API request.
103103
<sup>3</sup> Access to the chain-of-thought reasoning summary is limited access only for `o3` & `o4-mini`.
104104
<sup>4</sup> Streaming for `o3` is limited access only.
@@ -461,10 +461,13 @@ client = AzureOpenAI(
461461

462462
response = client.responses.create(
463463
input="Tell me about the curious case of neural text degeneration",
464-
model="o4-mini", # replace with model deployment name
464+
model="gpt-5", # replace with model deployment name
465465
reasoning={
466466
"effort": "medium",
467-
"summary": "detailed" # auto, concise, or detailed (currently only supported with o4-mini and o3)
467+
"summary": "auto" # auto, concise, or detailed
468+
}
469+
text={
470+
"verbosity": "low" # New with GPT-5 models
468471
}
469472
)
470473

@@ -478,47 +481,41 @@ curl -X POST "https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/responses?ap
478481
-H "Content-Type: application/json" \
479482
-H "Authorization: Bearer $AZURE_OPENAI_AUTH_TOKEN" \
480483
-d '{
481-
"model": "o4-mini",
484+
"model": "gpt-5",
482485
"input": "Tell me about the curious case of neural text degeneration",
483-
"reasoning": {"summary": "detailed"}
486+
"reasoning": {"summary": "auto"},
487+
"text": {"verbosity": "low"}
484488
}'
485489
```
486490

487491
---
488492

489493
```output
490494
{
491-
"id": "resp_68007e26b2cc8190b83361014f3a78c50ae9b88522c3ad24",
492-
"created_at": 1744862758.0,
495+
"id": "resp_689a0a3090808190b418acf12b5cc40e0fc1c31bc69d8719",
496+
"created_at": 1754925616.0,
493497
"error": null,
494498
"incomplete_details": null,
495499
"instructions": null,
496500
"metadata": {},
497-
"model": "o4-mini",
501+
"model": "gpt-5",
498502
"object": "response",
499503
"output": [
500504
{
501-
"id": "rs_68007e2773bc8190b5b8089949bfe13a0ae9b88522c3ad24",
502-
"summary": [
503-
{
504-
"text": "**Summarizing neural text degeneration**\n\nThe user's asking about \"The Curious Case of Neural Text Degeneration,\" a paper by Ari Holtzman et al. from 2020. It explains how certain decoding strategies produce repetitive and dull text. In contrast, methods like nucleus sampling yield more coherent and diverse outputs. The authors introduce metrics like surprisal and distinct-n for evaluation and suggest that maximum likelihood decoding often favors generic continuations, leading to loops and repetitive patterns in longer texts. They promote sampling from truncated distributions for improved text quality.",
505-
"type": "summary_text"
506-
},
507-
{
508-
"text": "**Explaining nucleus sampling**\n\nThe authors propose nucleus sampling, which captures a specified mass of the predictive distribution, improving metrics such as coherence and diversity. They identify a \"sudden drop\" phenomenon in token probabilities, where a few tokens dominate, leading to a long tail. By truncating this at a cumulative probability threshold, they aim to enhance text quality compared to top-k sampling. Their evaluations include human assessments, showing better results in terms of BLEU scores and distinct-n measures. Overall, they highlight how decoding strategies influence quality and recommend adaptive techniques for improved outcomes.",
509-
"type": "summary_text"
510-
}
511-
],
505+
"id": "rs_689a0a329298819095d90c34dc9b80db0fc1c31bc69d8719",
506+
"summary": [],
512507
"type": "reasoning",
508+
"encrypted_content": null,
513509
"status": null
514510
},
515511
{
516-
"id": "msg_68007e35c44881908cb4651b8e9972300ae9b88522c3ad24",
512+
"id": "msg_689a0a33009881909fe0fcf57cba30200fc1c31bc69d8719",
517513
"content": [
518514
{
519515
"annotations": [],
520-
"text": "Researchers first became aware that neural language models, when used to generate long stretches of text with standard “maximum‐likelihood” decoding (greedy search, beam search, etc.), often produce bland, repetitive or looping output. The 2020 paper “The Curious Case of Neural Text Degeneration” (Holtzman et al.) analyzes this failure mode and proposes a simple fix—nucleus (top‑p) sampling—that dramatically improves output quality.\n\n1. The Problem: Degeneration \n • With greedy or beam search, models tend to pick very high‑probability tokens over and over, leading to loops (“the the the…”) or generic, dull continuations. \n • Even sampling with a fixed top‑k (e.g. always sample from the 40 most likely tokens) can be suboptimal: if the model’s probability mass is skewed, k may be too small (overly repetitive) or too large (introducing incoherence).\n\n2. Why It Happens: Distributional Peakedness \n • At each time step the model’s predicted next‐token distribution often has one or two very high‑probability tokens, then a long tail of low‑probability tokens. \n • Maximum‐likelihood decoding zeroes in on the peak, collapsing diversity. \n • Uniform sampling over a large k allows low‑probability “wild” tokens, harming coherence.\n\n3. The Fix: Nucleus (Top‑p) Sampling \n • Rather than fixing k, dynamically truncate the distribution to the smallest set of tokens whose cumulative probability ≥ p (e.g. p=0.9). \n • Then renormalize and sample from that “nucleus.” \n • This keeps only the “plausible” mass and discards the improbable tail, adapting to each context.\n\n4. Empirical Findings \n • Automatic metrics (distinct‑n, repetition rates) and human evaluations show nucleus sampling yields more diverse, coherent, on‑topic text than greedy/beam or fixed top‑k. \n • It also outperforms simple temperature scaling (raising logits to 1/T) because it adapts to changes in the distribution’s shape.\n\n5. Takeaways for Practitioners \n • Don’t default to beam search for open-ended generation—its high likelihood doesn’t mean high quality. \n • Use nucleus sampling (p between 0.8 and 0.95) for a balance of diversity and coherence. \n • Monitor repetition and distinct‑n scores if you need automatic sanity checks.\n\nIn short, “neural text degeneration” is the tendency of likelihood‐maximizing decoders to produce dull or looping text. By recognizing that the shape of the model’s probability distribution varies wildly from step to step, nucleus sampling provides an elegant, adaptive way to maintain both coherence and diversity in generated text.",
521-
"type": "output_text"
516+
"text": "Neural text degeneration refers to the ways language models produce low-quality, repetitive, or vacuous text, especially when generating long outputs. It’s “curious” because models trained to imitate fluent text can still spiral into unnatural patterns. Key aspects:\n\n- Repetition and loops: The model repeats phrases or sentences (“I’m sorry, but...”), often due to high-confidence tokens reinforcing themselves.\n- Loss of specificity: Vague, generic, agreeable text that avoids concrete details.\n- Drift and contradiction: The output gradually departs from context or contradicts itself over long spans.\n- Exposure bias: During training, models see gold-standard prefixes; at inference, they must condition on their own imperfect outputs, compounding errors.\n- Likelihood vs. quality mismatch: Maximizing token-level likelihood doesn’t align with human preferences for diversity, coherence, or factuality.\n- Token over-optimization: Frequent, safe tokens get overused; certain phrases become attractors.\n- Entropy collapse: With greedy or low-temperature decoding, the distribution narrows too much, causing repetitive, low-entropy text.\n- Length and beam search issues: Larger beams or long generations can favor bland, repetitive sequences (the “likelihood trap”).\n\nCommon mitigations:\n\n- Decoding strategies:\n - Top-k, nucleus (top-p), or temperature sampling to keep sufficient entropy.\n - Typical sampling and locally typical sampling to avoid dull but high-probability tokens.\n - Repetition penalties, presence/frequency penalties, no-repeat n-grams.\n - Contrastive decoding (and variants like DoLa) to filter generic continuations.\n - Min/max length, stop sequences, and beam search with diversity/penalties.\n\n- Training and alignment:\n - RLHF/DPO to better match human preferences for non-repetitive, helpful text.\n - Supervised fine-tuning on high-quality, diverse data; instruction tuning.\n - Debiasing objectives (unlikelihood training) to penalize repetition and banned patterns.\n - Mixture-of-denoisers or latent planning to improve long-range coherence.\n\n- Architectural and planning aids:\n - Retrieval-augmented generation to ground outputs.\n - Tool use and structured prompting to constrain drift.\n - Memory and planning modules, hierarchical decoding, or sentence-level control.\n\n- Prompting tips:\n - Ask for concise answers, set token limits, and specify structure.\n - Provide concrete constraints or content to reduce generic filler.\n - Use “say nothing if uncertain” style instructions to avoid vacuity.\n\nRepresentative papers/terms to search:\n- Holtzman et al., “The Curious Case of Neural Text Degeneration” (2020): nucleus sampling.\n- Welleck et al., “Neural Text Degeneration with Unlikelihood Training.”\n- Li et al., “A Contrastive Framework for Decoding.”\n- Su et al., “DoLa: Decoding by Contrasting Layers.”\n- Meister et al., “Typical Decoding.”\n- Ouyang et al., “Training language models to follow instructions with human feedback.”\n\nIn short, degeneration arises from a mismatch between next-token likelihood and human preferences plus decoding choices; careful decoding, training objectives, and grounding help prevent it.",
517+
"type": "output_text",
518+
"logprobs": null
522519
}
523520
],
524521
"role": "assistant",
@@ -531,32 +528,40 @@ curl -X POST "https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/responses?ap
531528
"tool_choice": "auto",
532529
"tools": [],
533530
"top_p": 1.0,
531+
"background": false,
534532
"max_output_tokens": null,
533+
"max_tool_calls": null,
535534
"previous_response_id": null,
535+
"prompt": null,
536+
"prompt_cache_key": null,
536537
"reasoning": {
537-
"effort": "medium",
538+
"effort": "minimal",
538539
"generate_summary": null,
539540
"summary": "detailed"
540541
},
542+
"safety_identifier": null,
543+
"service_tier": "default",
541544
"status": "completed",
542545
"text": {
543546
"format": {
544547
"type": "text"
545548
}
546549
},
550+
"top_logprobs": null,
547551
"truncation": "disabled",
548552
"usage": {
549553
"input_tokens": 16,
550-
"output_tokens": 974,
551-
"output_tokens_details": {
552-
"reasoning_tokens": 384
553-
},
554-
"total_tokens": 990,
555554
"input_tokens_details": {
556555
"cached_tokens": 0
557-
}
556+
},
557+
"output_tokens": 657,
558+
"output_tokens_details": {
559+
"reasoning_tokens": 0
560+
},
561+
"total_tokens": 673
558562
},
559563
"user": null,
564+
"content_filters": null,
560565
"store": true
561566
}
562567
```
@@ -565,6 +570,152 @@ curl -X POST "https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/responses?ap
565570

566571
GPT-5 series reasoning models have the ability to call a new `custom_tool` called `lark_tool`. This tool is based on [Python lark](https://github.com/lark-parser/lark) and can be used for more flexible constraining of model output.
567572

573+
### Responses API
574+
575+
```json
576+
{
577+
"model": "gpt-5-2025-08-07",
578+
"input": "please calculate the area of a circle with radius equal to the number of 'r's in strawberry",
579+
"tools": [
580+
{
581+
"type": "custom",
582+
"name": "lark_tool",
583+
"format": {
584+
"type": "grammar",
585+
"syntax": "lark",
586+
"definition": "start: QUESTION NEWLINE ANSWER\nQUESTION: /[^\\n?]{1,200}\\?/\nNEWLINE: /\\n/\nANSWER: /[^\\n!]{1,200}!/"
587+
}
588+
}
589+
],
590+
"tool_choice": "required"
591+
}
592+
```
593+
594+
```python
595+
from openai import AzureOpenAI
596+
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
597+
598+
token_provider = get_bearer_token_provider(
599+
DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default"
600+
)
601+
602+
client = AzureOpenAI(
603+
base_url = "https://YOUR-RESOURCE-NAME-HERE.openai.azure.com/openai/v1/",
604+
azure_ad_token_provider=token_provider,
605+
api_version="preview"
606+
)
607+
608+
response = client.responses.create(
609+
model="gpt-5", # replace with your model deployment name
610+
tools=[
611+
{
612+
"type": "custom",
613+
"name": "lark_tool",
614+
"format": {
615+
"type": "grammar",
616+
"syntax": "lark",
617+
"definition": "start: QUESTION NEWLINE ANSWER\nQUESTION: /[^\\n?]{1,200}\\?/\nNEWLINE: /\\n/\nANSWER: /[^\\n!]{1,200}!/"
618+
}
619+
}
620+
],
621+
input=[{"role": "user", "content": "Please calculate the area of a circle with radius equal to the number of 'r's in strawberry"}],
622+
)
623+
624+
print(response.model_dump_json(indent=2))
625+
626+
```
627+
628+
**Output**:
629+
630+
```json
631+
{
632+
"id": "resp_689a0cf927408190b8875915747667ad01c936c6ffb9d0d3",
633+
"created_at": 1754926332.0,
634+
"error": null,
635+
"incomplete_details": null,
636+
"instructions": null,
637+
"metadata": {},
638+
"model": "gpt-5",
639+
"object": "response",
640+
"output": [
641+
{
642+
"id": "rs_689a0cfd1c888190a2a67057f471b5cc01c936c6ffb9d0d3",
643+
"summary": [],
644+
"type": "reasoning",
645+
"encrypted_content": null,
646+
"status": null
647+
},
648+
{
649+
"id": "msg_689a0d00e60c81908964e5e9b2d6eeb501c936c6ffb9d0d3",
650+
"content": [
651+
{
652+
"annotations": [],
653+
"text": "“strawberry” has 3 r’s, so the radius is 3.\nArea = πr² = π × 3² = 9π ≈ 28.27 square units.",
654+
"type": "output_text",
655+
"logprobs": null
656+
}
657+
],
658+
"role": "assistant",
659+
"status": "completed",
660+
"type": "message"
661+
}
662+
],
663+
"parallel_tool_calls": true,
664+
"temperature": 1.0,
665+
"tool_choice": "auto",
666+
"tools": [
667+
{
668+
"name": "lark_tool",
669+
"parameters": null,
670+
"strict": null,
671+
"type": "custom",
672+
"description": null,
673+
"format": {
674+
"type": "grammar",
675+
"definition": "start: QUESTION NEWLINE ANSWER\nQUESTION: /[^\\n?]{1,200}\\?/\nNEWLINE: /\\n/\nANSWER: /[^\\n!]{1,200}!/",
676+
"syntax": "lark"
677+
}
678+
}
679+
],
680+
"top_p": 1.0,
681+
"background": false,
682+
"max_output_tokens": null,
683+
"max_tool_calls": null,
684+
"previous_response_id": null,
685+
"prompt": null,
686+
"prompt_cache_key": null,
687+
"reasoning": {
688+
"effort": "medium",
689+
"generate_summary": null,
690+
"summary": null
691+
},
692+
"safety_identifier": null,
693+
"service_tier": "default",
694+
"status": "completed",
695+
"text": {
696+
"format": {
697+
"type": "text"
698+
}
699+
},
700+
"top_logprobs": null,
701+
"truncation": "disabled",
702+
"usage": {
703+
"input_tokens": 139,
704+
"input_tokens_details": {
705+
"cached_tokens": 0
706+
},
707+
"output_tokens": 240,
708+
"output_tokens_details": {
709+
"reasoning_tokens": 192
710+
},
711+
"total_tokens": 379
712+
},
713+
"user": null,
714+
"content_filters": null,
715+
"store": true
716+
}
717+
```
718+
568719
### Chat Completions
569720

570721
```json
@@ -596,30 +747,6 @@ GPT-5 series reasoning models have the ability to call a new `custom_tool` calle
596747
}
597748
```
598749

599-
### Responses API
600-
601-
```
602-
{
603-
"model": "gpt-5-2025-08-07",
604-
"input": "please calculate the area of a circle with radius equal to the number of 'r's in strawberry",
605-
"tools": [
606-
{
607-
"type": "custom",
608-
"name": "lark_tool",
609-
"format": {
610-
"type": "grammar",
611-
"syntax": "lark",
612-
"definition": "start: QUESTION NEWLINE ANSWER\nQUESTION: /[^\\n?]{1,200}\\?/\nNEWLINE: /\\n/\nANSWER: /[^\\n!]{1,200}!/"
613-
}
614-
}
615-
],
616-
"tool_choice": "required"
617-
}
618-
```
619-
620-
621-
622-
623750
## Markdown output
624751

625752
By default the `o3-mini` and `o1` models will not attempt to produce output that includes markdown formatting. A common use case where this behavior is undesirable is when you want the model to output code contained within a markdown code block. When the model generates output without markdown formatting you lose features like syntax highlighting, and copyable code blocks in interactive playground experiences. To override this new default behavior and encourage markdown inclusion in model responses, add the string `Formatting re-enabled` to the beginning of your developer message.

0 commit comments

Comments
 (0)