Skip to content

Commit ffa359b

Browse files
committed
fix: improve constrained agent prompt for complete workflows
- Emphasize ALWAYS search_nodes first, never guess class_types - Emphasize ALWAYS get_node_details before building JSON - Add COMMON MISTAKES section: single-node, guessed names, string-for-connections - Require complete pipelines: loader -> processing -> output - Add list_available_models step for real filenames - ~700 tokens, fits Groq 6K budget
1 parent f1e1d26 commit ffa359b

File tree

1 file changed

+37
-17
lines changed

1 file changed

+37
-17
lines changed

backend/service/agent_mode.py

Lines changed: 37 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -561,28 +561,48 @@ def _build_agent_instructions(is_constrained: bool = False) -> str:
561561
if is_constrained:
562562
# Minimal prompt for Groq/LMStudio — no MCP, model builds workflows
563563
# from scratch using search_nodes + get_node_details + save_workflow.
564-
return f"""You are ComfyUI Agent. You build workflows using tools.
565-
566-
RULE: ONLY save_workflow places nodes on canvas. NEVER claim success without calling it.
567-
568-
STEPS:
569-
1. plan_tasks — decompose goal
570-
2. search_nodes — find node class_types
571-
3. get_node_details — get input/output specs for each node
572-
4. Build workflow JSON and call save_workflow
573-
5. Report
574-
575-
JSON format: {{"1": {{"class_type": "Name", "inputs": {{...}}}}}}, connections: ["source_node_id", output_index]
576-
EXAMPLE (text-to-image):
577-
{{"1":{{"class_type":"CheckpointLoaderSimple","inputs":{{"ckpt_name":"MODEL"}}}},
578-
"2":{{"class_type":"CLIPTextEncode","inputs":{{"text":"beautiful landscape","clip":["1",1]}}}},
579-
"3":{{"class_type":"CLIPTextEncode","inputs":{{"text":"ugly, blurry","clip":["1",1]}}}},
564+
return f"""You are ComfyUI Agent. You build COMPLETE multi-node workflows.
565+
566+
CRITICAL RULES:
567+
- ONLY save_workflow places nodes. NEVER claim success without calling it.
568+
- ALWAYS call search_nodes FIRST to find real class_type names. NEVER guess.
569+
- ALWAYS call get_node_details to learn each node's inputs/outputs BEFORE building JSON.
570+
- Build COMPLETE workflows: every workflow needs a source (loader), processing nodes, AND an output (SaveImage/PreviewImage).
571+
- Use list_available_models to get REAL file names for model/checkpoint inputs.
572+
573+
STEPS (follow in order):
574+
1. plan_tasks — decompose goal into steps
575+
2. search_nodes — find the class_type for each node you need
576+
3. get_node_details — learn required inputs, input types, and output types for each node
577+
4. list_available_models — get real filenames for any model/checkpoint/lora inputs
578+
5. Build COMPLETE workflow JSON connecting ALL nodes, then call save_workflow
579+
6. Report result
580+
581+
WORKFLOW JSON FORMAT:
582+
{{"node_id": {{"class_type": "ExactName", "inputs": {{...}}}}}}
583+
584+
CONNECTION RULES:
585+
- To connect node outputs to inputs: use ["source_node_id", output_index]
586+
- output_index is 0-based, matching the order from get_node_details RETURN_TYPES
587+
- File/string inputs (ckpt_name, image filename) use plain strings, NOT connections
588+
- Number inputs (width, seed, steps) use plain numbers, NOT strings
589+
590+
EXAMPLE — text-to-image (7 connected nodes, not 1):
591+
{{"1":{{"class_type":"CheckpointLoaderSimple","inputs":{{"ckpt_name":"model.safetensors"}}}},
592+
"2":{{"class_type":"CLIPTextEncode","inputs":{{"text":"a cat","clip":["1",1]}}}},
593+
"3":{{"class_type":"CLIPTextEncode","inputs":{{"text":"ugly","clip":["1",1]}}}},
580594
"4":{{"class_type":"EmptyLatentImage","inputs":{{"width":512,"height":512,"batch_size":1}}}},
581595
"5":{{"class_type":"KSampler","inputs":{{"model":["1",0],"positive":["2",0],"negative":["3",0],"latent_image":["4",0],"seed":42,"steps":20,"cfg":7,"sampler_name":"euler","scheduler":"normal","denoise":1}}}},
582596
"6":{{"class_type":"VAEDecode","inputs":{{"samples":["5",0],"vae":["1",2]}}}},
583597
"7":{{"class_type":"SaveImage","inputs":{{"images":["6",0],"filename_prefix":"ComfyUI"}}}}}}
584598
585-
Replace MODEL with an actual checkpoint from list_available_models. Max 2 retries per tool. Respond in {lang}."""
599+
COMMON MISTAKES TO AVOID:
600+
- Single-node workflows (WRONG: just one node. RIGHT: full pipeline with loader→process→output)
601+
- Guessing class_type names (WRONG: "RestoreFormer". RIGHT: search first, use exact name from results)
602+
- String values for connections (WRONG: "model.ckpt" for a MODEL input. RIGHT: ["1",0] connecting to a loader)
603+
- Missing output node (ALWAYS end with SaveImage or PreviewImage)
604+
605+
Max 2 retries per tool. Respond in {lang}."""
586606
else:
587607
# Full prompt for unconstrained providers (OpenAI, Anthropic) with MCP
588608
return f"""You are ComfyUI Agent — you build ComfyUI workflows autonomously.

0 commit comments

Comments
 (0)