Dynamic Workflows: Make Nodes Primary and Generate Python Code

[ajbozarth]

Note: This issue description was AI-generated based on internal issue #108. Please review, validate, and adjust the details as needed.

Problem Statement

Currently, Qiskit Studio uses Python code as the source of truth, with GUI nodes mapping to specific parts of that code. When parameters change in a node, the application uses AI to update the corresponding Python code. This approach has limitations:

1. Changes to the Python code don't automatically update the node representation
2. Complex code may not map cleanly to the visual representation
3. The node graph doesn't fully represent the quantum workflow's state
4. Users must understand both the node interface and the Python code

We need to implement Dynamic Workflows: nodes should be the primary source of truth, and Python code should be generated from the nodes.

Current Implementation

• Python code is the source of truth for quantum workflows
• Nodes in the GUI map to specific parts of the Python code
• When parameters change in a node, AI generates updated Python code
• The code generation flow is:
  • User modifies node → AI updates Python code → Code displayed in panel
• Key components:
  • lib/code-generator.ts: Generates Python code from nodes
  • lib/api-service.ts: Handles AI code generation
  • hooks/useAICodeGeneration.ts: React hook for AI code generation
  • components/quantum-composer.tsx: Manages nodes and edges
  • components/nodes/: Individual node implementations

Desired Implementation

• Dynamic Workflows: Nodes become the primary source of truth for quantum workflows
• Each node stores its complete configuration and state
• Python code is generated from the nodes' configurations
• The code generation flow becomes:
  • User modifies node → Node state updates → Python code generated → Code displayed in panel
• Bidirectional synchronization is supported, but nodes remain the primary source of truth
• Code changes can still update node configurations when appropriate
• The node graph fully represents the quantum workflow

Implementation Approach

1. Enhanced Node Data Model:

   • Expand node data structure to store all necessary configuration
   • Add validation methods to ensure node configurations are valid
   • Implement methods to generate Python code from node state

2. Refactored Code Generation:

   • Update lib/code-generator.ts to generate code primarily from node configurations
   • Implement bidirectional synchronization to allow code changes to update nodes when appropriate
   • Implement a more robust templating system for code generation

3. Updated AI Integration:

   • Modify AI integration to focus on enhancing node configurations
   • Update prompts to work with node-centric approach
   • Use AI to suggest node configurations rather than modifying code directly

4. Custom Code Support:

   • Add specialized "Python" nodes for custom code sections
   • Implement a mechanism to incorporate custom code into the generated output
   • Ensure custom code sections are preserved during code generation

5. Migration Strategy:

   • Develop a migration path for existing workflows
   • Support both approaches during a transition period
   • Create tools to convert Python-based workflows to node-based workflows

Benefits

1. Improved User Experience:

   • Users can focus on the visual interface without needing to understand Python
   • Changes to the workflow are immediately reflected in the node graph
   • More intuitive representation of quantum workflows

2. Better Consistency:

   • No risk of divergence between node representation and code
   • Guaranteed that the node graph represents the full workflow
   • More predictable behavior when modifying workflows

3. Enhanced Extensibility:

   • Easier to add new node types and functionality
   • More modular architecture with clearer separation of concerns
   • Better foundation for future collaborative features

4. Simplified Architecture:

   • Clearer data flow from nodes to code
   • Reduced complexity in state management
   • More maintainable codebase

Potential Challenges

1. Complex Code Representation:

   • Some quantum operations might be difficult to represent purely through nodes
   • Need to ensure all Qiskit functionality can be represented in the node system

2. Backward Compatibility:

   • Existing workflows are based on Python code as the source of truth
   • Need a migration strategy for existing content

3. AI Integration:

   • Current AI code generation is designed to update Python code
   • Need to adapt the AI integration to work with nodes as the primary source of truth
   • Support bidirectional updates where AI can suggest node configuration changes based on code

4. Custom Code Support:

   • Users may want to add custom Python code that doesn't fit into the node system
   • Need a way to incorporate custom code while maintaining nodes as the primary source of truth

5. Performance:

   • Generating code from a complex node graph could be computationally intensive
   • Need to ensure good performance even with large workflows

Implementation Steps

1. Analysis and Planning:

   • Audit existing node types and their parameters
   • Define the complete data model for each node type
   • Create a detailed migration plan

2. Core Architecture Updates:

   • Enhance node data structures to store complete configuration
   • Update node components to manage their own state
   • Refactor code generation to work with the new node model

3. AI Integration Updates:

   • Modify AI service integration to work with node configurations
   • Update prompts and response handling
   • Implement new AI-assisted features for node configuration

4. Custom Code Support:

   • Implement Python nodes for custom code sections
   • Develop mechanisms to incorporate custom code into generated output
   • Create validation tools for custom code

5. Migration Tools:

   • Develop tools to convert existing Python-based workflows to node-based
   • Create documentation for migration process
   • Implement backward compatibility features

6. Testing and Validation:

   • Develop comprehensive tests for the new architecture
   • Validate generated code against expected output
   • Test migration of existing workflows

7. Documentation and Training:

   • Update documentation to reflect the new architecture
   • Create tutorials for the node-centric approach
   • Provide migration guides for existing users

Technical Debt Considerations

This architectural change will address several areas of technical debt:

1. The current dual source of truth creates complexity and potential inconsistencies
2. The tight coupling between code and nodes makes the system harder to maintain
3. The current approach limits the potential for future collaborative features
4. The reliance on AI for code updates introduces unpredictability

By implementing Dynamic Workflows with nodes as the primary source of truth, we'll create a more maintainable, extensible, and user-friendly system.

Questions for Discussion

1. Should we support both approaches during a transition period, or make a clean break?
2. How should we handle complex Python code that doesn't map cleanly to nodes?
3. What level of backward compatibility do we need to maintain?
4. How should we prioritize node types for the migration?
5. What new node types might be needed to support all Qiskit functionality?

Files to Modify

• lib/code-generator.ts - Refactor to generate from node state
• lib/api-service.ts - Update AI integration
• hooks/useAICodeGeneration.ts - Adapt for node-centric approach
• components/quantum-composer.tsx - Update state management
• components/nodes/*.tsx - Enhance node data models
• components/nodes/python-node.tsx - Add/enhance for custom code

Acceptance Criteria

• Node data structures enhanced to store complete configuration
• Code generation works primarily from node configurations
• Bidirectional synchronization implemented
• Python nodes support custom code sections
• Migration tools created for existing workflows
• Documentation updated
• Tests cover new architecture
• Performance validated with complex workflows