Adding Control Flow Functionality to Workflows

[jackpeplinski]

Current Limitations

Currently, workflow steps are recorded in JSON. While JSON is a convenient data exchange format, it lacks native control flow functionality, making it difficult to express common patterns such as:

• Iterating over multiple elements (e.g., clicking a download button for every file on a page)
• Conditional execution (e.g., only click if an element exists)

Options

Option: AI-Generated Code
While generating code and just limiting the functions accessible would be optimal in my view, this quickly becomes complex since no current LLMs verify the generated code, AFAIK. e.g., If I prompt "Only use functions A, B, C... while writing your code" there's no guarantee these functions 1) aren't in the code and 2) that the code is structured as I intend.

Option: JSON-Based
An alternative is to add a few common control flow steps to the JSON schema since many LLMs do provide structured outputs, which can then be transformed into Python code. Users could modify either the JSON or the generated Python, depending on their preferences. This also allows for more control when a workflow pattern cannot currently be expressed in the JSON schema.

Implementation

Proposed Initial JSON Extensions

• ForLoopStep / WhileLoopStep for iteration
• ConditionalStep for if_exists and other conditional logic
• Loop variables ({index}, {item}) and scoping for dynamic execution

Implementation Outline

1. Extended JSON Schema – Add new step types (ForLoopStep, WhileLoopStep, ConditionalStep) in views.py
2. JSON-to-Python Code Generator – Convert control flow JSON to executable Python with proper indentation in workflows/workflow_use/codegen/generator.py
3. Workflow Execution Engine – Extend Workflow class in service.py to handle loops and context
4. Builder Service Updates – Update LLM prompts to generate control flow structures in prompts.py
5. Context Management – Support loop variables and scoping in service.py

Conclusion

This approach keeps the structured JSON format while enabling richer workflow logic. Let me know your thoughts.

Note
P.S., While exploring this idea, I noticed Skyvern is pursuing a similar idea, though they currently lack a browser extension recording setup and I find their documentation less up-to-date.